Skip to content

feat(documents): add a streaming document export - #158

Open
tharropoulos wants to merge 2 commits into
typesense:masterfrom
tharropoulos:feat/export-stream
Open

tharropoulos wants to merge 2 commits into
typesense:masterfrom
tharropoulos:feat/export-stream

Conversation

@tharropoulos

Copy link
Copy Markdown
Collaborator

Change Summary

  • add documents.export_stream(), a single-use context manager that yields raw jsonl lines or decoded bytes
  • split lines on the \n byte, since iter_lines breaks documents containing U+2028, U+2029 or U+0085
  • release the max_concurrent_requests slot at headers and limit open exports to max_connections - max_concurrent_requests
  • allow one export when that limit is not positive, and skip the limit for a caller's http_client or an unlimited pool
  • request Accept-Encoding: identity unless compress=True, keeping any value set in additional_headers
  • raise on reads after a close, a failed read or an invalid utf-8 line, so a partial export cannot look complete
  • closes Feature request: opt-in incremental JSONL document export #157

PR Checklist

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: opt-in incremental JSONL document export

1 participant