-
Notifications
You must be signed in to change notification settings - Fork 120
Enhance SEA HTTP Client #618
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
453 commits
Select commit
Hold shift + click to select a range
bf6d41c
ensure command_id is not None
varun-edachali-dbx 5afa733
line breaks after multi-line pyfocs
varun-edachali-dbx e3dfd36
ensure non null operationHandle for commandId creation
varun-edachali-dbx 63360b3
use command_id methods instead of explicit guid_to_hex_id conversion
varun-edachali-dbx 13ffb8d
remove un-necessary artifacts in test_session, add back assertion
varun-edachali-dbx a74d279
Implement SeaDatabricksClient (Complete Execution Spec) (#590)
varun-edachali-dbx ffd478e
Merge branch 'sea-migration' into metadata-sea
varun-edachali-dbx f6d873d
remove unimplemented metadata methods test, un-necessary imports
varun-edachali-dbx 28675f5
introduce unit tests for metadata methods
varun-edachali-dbx d759050
add from __future__ import annotations to remove string literals arou…
varun-edachali-dbx 3578659
remove verbosity in ResultSetFilter docstring
varun-edachali-dbx 8713023
remove un-necessary info in ResultSetFilter docstring
varun-edachali-dbx 22dc252
remove explicit type checking, string literals around forward annotat…
varun-edachali-dbx 390f592
house SQL commands in constants
varun-edachali-dbx dd7dc6a
convert complex types to string if not _use_arrow_native_complex_types
varun-edachali-dbx 28308fe
Merge branch 'metadata-sea' into fetch-json-inline
varun-edachali-dbx 2712d1c
introduce unit tests for altered functionality
varun-edachali-dbx dabba55
Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbx 48ad7b3
Revert "Merge branch 'fetch-json-inline' into ext-links-sea"
varun-edachali-dbx a1f9b9c
reduce verbosity of ResultSetFilter docstring
varun-edachali-dbx 984e8ee
remove unused imports
varun-edachali-dbx 3a999c0
Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbx c313c2b
Revert "Merge branch 'fetch-json-inline' into ext-links-sea"
varun-edachali-dbx 3bc615e
Revert "reduce verbosity of ResultSetFilter docstring"
varun-edachali-dbx b6e1a10
Reapply "Merge branch 'fetch-json-inline' into ext-links-sea"
varun-edachali-dbx 2df3d39
Revert "Merge branch 'fetch-json-inline' into ext-links-sea"
varun-edachali-dbx 5e75fb5
remove un-necessary filters changes
varun-edachali-dbx 20822e4
remove un-necessary backend changes
varun-edachali-dbx 802d045
remove constants changes
varun-edachali-dbx f3f795a
remove changes in filters tests
varun-edachali-dbx f6c5950
remove unit test backend and JSON queue changes
varun-edachali-dbx d210ccd
remove changes in sea result set testing
varun-edachali-dbx 22a953e
Revert "remove changes in sea result set testing"
varun-edachali-dbx 3aed144
Revert "remove unit test backend and JSON queue changes"
varun-edachali-dbx 0fe4da4
Revert "remove changes in filters tests"
varun-edachali-dbx 0e3c0a1
Revert "remove constants changes"
varun-edachali-dbx 93edb93
Revert "remove un-necessary backend changes"
varun-edachali-dbx 871a44f
Revert "remove un-necessary filters changes"
varun-edachali-dbx 0ce144d
remove unused imports
varun-edachali-dbx 08ca60d
Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbx 8c5cc77
working version
varun-edachali-dbx 7f5c715
adopy _wait_until_command_done
varun-edachali-dbx 9ef5fad
introduce metadata commands
varun-edachali-dbx 44183db
use new backend structure
varun-edachali-dbx d59b351
constrain backend diff
varun-edachali-dbx 1edc80a
remove changes to filters
varun-edachali-dbx f82658a
make _parse methods in models internal
varun-edachali-dbx 54eb0a4
reduce changes in unit tests
varun-edachali-dbx 50cc1e2
run small queries with SEA during integration tests
varun-edachali-dbx 242307a
run some tests for sea
varun-edachali-dbx 1e21434
move docstring of DatabricksClient within class
varun-edachali-dbx cd4015b
move ThriftResultSet import to top of file
varun-edachali-dbx ed8b610
make backend/utils __init__ file empty
varun-edachali-dbx 94d951e
use from __future__ import annotations to remove string literals arou…
varun-edachali-dbx c20058e
use lazy logging
varun-edachali-dbx fe3acb1
replace getters with property tag
varun-edachali-dbx 9fb6a76
Merge branch 'main' into backend-refactors
varun-edachali-dbx 61dfc4d
set active_command_id to None, not active_op_handle
varun-edachali-dbx 64fb9b2
align test_session with pytest instead of unittest
varun-edachali-dbx 8a138e8
allow empty schema bytes for alignment with SEA
varun-edachali-dbx 82f9d6b
pass is_vl_op to Sea backend ExecuteResponse
varun-edachali-dbx 35f1ef0
remove catalog requirement in get_tables
varun-edachali-dbx a515d26
move filters.py to SEA utils
varun-edachali-dbx 59b1330
ensure SeaResultSet
varun-edachali-dbx cbf63f9
Merge branch 'main' into sea-migration
varun-edachali-dbx 59b4825
remove duplicate test, correct active_command_id attribute
varun-edachali-dbx 293e356
Merge branch 'sea-migration' into metadata-sea
varun-edachali-dbx dd40beb
prevent circular imports
varun-edachali-dbx 14057ac
remove unused imports
varun-edachali-dbx a4d5bdb
remove cast, throw error if not SeaResultSet
varun-edachali-dbx 156421a
Merge branch 'metadata-sea' into fetch-json-inline
varun-edachali-dbx eb1a9b4
pass param as TSparkParameterValue
varun-edachali-dbx 9000666
Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbx a3ca7c7
remove failing test (temp)
varun-edachali-dbx 2c22010
remove SeaResultSet type assertion
varun-edachali-dbx c09508e
change errors to align with spec, instead of arbitrary ValueError
varun-edachali-dbx e9b1314
make SEA backend methods return SeaResultSet
varun-edachali-dbx 8ede414
use spec-aligned Exceptions in SEA backend
varun-edachali-dbx 09a1b11
remove defensive row type check
varun-edachali-dbx e380654
SeaDatabricksClient: Add Metadata Commands (#593)
varun-edachali-dbx 677a7b0
SEA volume operations fix: assign `manifest.is_volume_operation` to `…
varun-edachali-dbx 5e01e7b
Merge branch 'metadata-sea' into fetch-json-inline
varun-edachali-dbx 3becefe
Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbx a026d31
raise ProgrammingError for invalid id
varun-edachali-dbx 4446a9e
make is_volume_operation strict bool
varun-edachali-dbx 138359d
remove complex types code
varun-edachali-dbx b99d0c4
Revert "remove complex types code"
varun-edachali-dbx 21c389d
introduce type conversion for primitive types for JSON + INLINE
varun-edachali-dbx 734321a
Merge branch 'sea-migration' into fetch-json-inline
varun-edachali-dbx 9f0f969
remove SEA running on metadata queries (known failures
varun-edachali-dbx 04a1936
remove un-necessary docstrings
varun-edachali-dbx 11bc165
preliminary (robust) SEA HTTP Client
varun-edachali-dbx d389316
prevent catching of MaxRetryError and HttpError in client
varun-edachali-dbx cc48caf
formatting (black)
varun-edachali-dbx 6a1274f
fix type annotations
varun-edachali-dbx 278b8cd
align expected types with databricks sdk
varun-edachali-dbx 91b7f7f
link rest api reference to validate types
varun-edachali-dbx 7a5ae13
remove test_catalogs_returns_arrow_table test
varun-edachali-dbx f1776f3
fix fetchall_arrow and fetchmany_arrow
varun-edachali-dbx 6143331
remove thrift aligned test_cancel_during_execute from SEA tests
varun-edachali-dbx 45585d4
Introduce manual SEA test scripts for Exec Phase (#589)
varun-edachali-dbx 8949d0c
Merge branch 'sea-migration' into fetch-json-inline
varun-edachali-dbx 5eaded4
remove un-necessary changes in example scripts
varun-edachali-dbx eeed9a1
remove un-necessary chagnes in example scripts
varun-edachali-dbx f233886
_convert_json_table -> _create_json_table
varun-edachali-dbx 68ac437
remove accidentally removed test
varun-edachali-dbx 7fd0845
remove new unit tests (to be re-added based on new arch)
varun-edachali-dbx ea7ff73
remove changes in sea_result_set functionality (to be re-added)
varun-edachali-dbx 563da71
introduce more integration tests
varun-edachali-dbx a018273
remove SEA tests in parameterized queries
varun-edachali-dbx c0e98f4
remove partial parameter fix changes
varun-edachali-dbx 7343035
remove un-necessary timestamp tests
varun-edachali-dbx ec500b6
slightly stronger typing of _convert_json_types
varun-edachali-dbx 0b3e91d
stronger typing of json utility func s
varun-edachali-dbx 7664e44
stronger typing of fetch*_json
varun-edachali-dbx db7b8e5
remove unused helper methods in SqlType
varun-edachali-dbx f75f2b5
line breaks after multi line pydocs, remove excess logs
varun-edachali-dbx e2d4ef5
line breaks after multi line pydocs, reduce diff of redundant changes
varun-edachali-dbx 21e3078
reduce diff of redundant changes
varun-edachali-dbx bb015e6
mandate ResultData in SeaResultSet constructor
varun-edachali-dbx 3944e39
Merge branch 'fetch-json-inline' into ext-links-sea
varun-edachali-dbx b3273c7
remove complex type conversion
varun-edachali-dbx 38c2b88
correct fetch*_arrow
varun-edachali-dbx ebc1915
Merge branch 'ext-links-sea' into sea-http-client
varun-edachali-dbx 4651cd6
pass test_retry_exponential_backoff
varun-edachali-dbx d67eb7b
prevent parsing empty response data (get test_retry_abort_non_recover…
varun-edachali-dbx 2caf38d
more defensive parsing, allow more method types in urllib3
varun-edachali-dbx 3e55ddd
allow Any values in session_conf, cast to String as done in Thrift ba…
varun-edachali-dbx 4afff39
account for max_redirects in SEA backend
varun-edachali-dbx 01d49cd
return empty JsonQueue if no data
varun-edachali-dbx 3d8aa7f
do not preload content?
varun-edachali-dbx 70c7dc8
Complete Fetch Phase (for `INLINE` disposition and `JSON_ARRAY` forma…
varun-edachali-dbx b77acbe
Merge branch 'sea-migration' into ext-links-sea
varun-edachali-dbx fa2359d
recover old sea tests
varun-edachali-dbx c07f709
move queue and result set into SEA specific dir
varun-edachali-dbx 9e4ef2e
pass ssl_options into CloudFetchQueue
varun-edachali-dbx b00c06c
reduce diff
varun-edachali-dbx 10f55f0
remove redundant conversion.py
varun-edachali-dbx cd119e9
fix type issues
varun-edachali-dbx d79638b
ValueError not ProgrammingError
varun-edachali-dbx f84578a
reduce diff
varun-edachali-dbx c621c0c
introduce SEA cloudfetch e2e tests
varun-edachali-dbx 7958cd9
allow empty cloudfetch result
varun-edachali-dbx e2d17ff
add unit tests for CloudFetchQueue and SeaResultSet
varun-edachali-dbx d348b35
skip pyarrow dependent tests
varun-edachali-dbx abf9aab
Merge branch 'main' into sea-migration
varun-edachali-dbx 811205e
Merge branch 'sea-migration' into ext-links-sea
varun-edachali-dbx 9b4b606
Merge branch 'main' into backend-refactors
varun-edachali-dbx 4bd290e
simplify download process: no pre-fetching
varun-edachali-dbx dfbbf79
correct class name in logs
varun-edachali-dbx b12a8b0
Merge branch 'ext-links-sea' into sea-http-client
varun-edachali-dbx 4f11ff0
Introduce `row_limit` param (#607)
varun-edachali-dbx ed4d7ab
Merge branch 'sea-migration' into ext-links-sea
varun-edachali-dbx 51aa9be
add sea tag on large queries
varun-edachali-dbx a5e9cdf
align with old impl
varun-edachali-dbx be16634
align next_n_rows with prev imple
varun-edachali-dbx 6ec8656
align remaining_rows with prev impl
varun-edachali-dbx 7ea7b75
remove un-necessary Optional params
varun-edachali-dbx 64be07b
remove un-necessary changes in thrift field if tests
varun-edachali-dbx 165644c
remove unused imports
varun-edachali-dbx 3172ab8
Merge branch 'ext-links-sea' into sea-http-client
varun-edachali-dbx 461e762
simplify error handling
varun-edachali-dbx fd1e6cf
stop mypy complaints
varun-edachali-dbx abef941
run large queries
varun-edachali-dbx 15378de
run retry tests
varun-edachali-dbx 83de7d9
Merge branch 'ext-links-sea' into sea-http-client
varun-edachali-dbx 45f5c26
Merge branch 'main' into backend-refactors
varun-edachali-dbx 2c9368a
formatting (black)
varun-edachali-dbx 9b1b1f5
remove repetition from Session.__init__
varun-edachali-dbx fce324b
move link fetching immediately before table creation so link expiry i…
varun-edachali-dbx 77e23d3
Merge branch 'backend-refactors' into sea-migration
varun-edachali-dbx 3bd3aef
fix merge artifacts
varun-edachali-dbx 6d4701f
correct patch paths
varun-edachali-dbx 39469fa
Merge branch 'sea-migration' into ext-links-sea
varun-edachali-dbx b2d1579
formatting (black)
varun-edachali-dbx 077a71c
fix types
varun-edachali-dbx 7985639
fix param type in unit tests
varun-edachali-dbx dc1cb6d
fix type issues
varun-edachali-dbx dfc32b4
Merge branch 'sea-migration' into ext-links-sea
varun-edachali-dbx 0a0643b
correct param extraction
varun-edachali-dbx f7fd1d9
remove common constructor for databricks client abc
varun-edachali-dbx 510b0a3
make SEA Http Client instance a private member
varun-edachali-dbx dd2864b
make GetChunksResponse model more robust
varun-edachali-dbx c32b281
add link to doc of GetChunk response model
varun-edachali-dbx 0b1eba5
pass result_data instead of "initial links" into SeaCloudFetchQueue
varun-edachali-dbx 777f7c1
move download_manager init into parent CloudFetchQueue
varun-edachali-dbx 130b0d3
raise ServerOperationError for no 0th chunk
varun-edachali-dbx 1920375
unused iports
varun-edachali-dbx 5d04cd0
Merge branch 'main' into sea-migration
varun-edachali-dbx 922c448
explicitly close result queue
varun-edachali-dbx d882c6e
Merge branch 'sea-migration' into ext-links-sea
varun-edachali-dbx 7fb59e4
Merge branch 'ext-links-sea' into sea-http-client
varun-edachali-dbx 25596e7
simplify HTTP client using requests.session
varun-edachali-dbx 2124fe0
explicitly close response object
varun-edachali-dbx 404aadf
formatting (black)
varun-edachali-dbx 01c9b95
remove excess changes
varun-edachali-dbx 5a43686
return None in case of empty respose
varun-edachali-dbx 28c6bb1
ensure table is empty on no initial link s
varun-edachali-dbx 41f1130
Merge branch 'ext-links-sea' into sea-http-client
varun-edachali-dbx 1a0575a
Complete Fetch Phase (`EXTERNAL_LINKS` disposition and `ARROW` format…
varun-edachali-dbx c07beb1
SEA Session Configuration Fix: Explicitly convert values to `str` (#…
varun-edachali-dbx 97707f5
Merge branch 'sea-migration' into sea-http-client
varun-edachali-dbx 36daee6
Revert "explicitly close response object"
varun-edachali-dbx b3fb5fa
Revert "simplify HTTP client using requests.session"
varun-edachali-dbx 7143eff
pass error as 4th arg, not 3rd
varun-edachali-dbx e3528a1
add warning for legacy behaviour
varun-edachali-dbx 112958d
simplify error handling (attempt)
varun-edachali-dbx b7a4677
cleaner HTTP client using requests.sessions
varun-edachali-dbx 640cc82
SEA: add support for `Hybrid` disposition (#631)
varun-edachali-dbx 3b8e1ad
Revert "cleaner HTTP client using requests.sessions"
varun-edachali-dbx c227739
clean up response (implicitly by using with)
varun-edachali-dbx a0aa023
improve docstrings, remove redundant comments and imports
varun-edachali-dbx ef9283b
unit tests for imp methods
varun-edachali-dbx 8fbca9d
SEA: Reduce network calls for synchronous commands (#633)
varun-edachali-dbx d8f46cb
Merge branch 'sea-migration' into sea-http-client
varun-edachali-dbx 806e5f5
SEA: Decouple Link Fetching (#632)
varun-edachali-dbx b57c3f3
Chunk download latency (#634)
saishreeeee ef5836b
acquire lock before notif + formatting (black)
varun-edachali-dbx 4fd2a3f
Merge branch 'main' into sea-migration
varun-edachali-dbx 26f8947
fix imports
varun-edachali-dbx 2d44596
add get_chunk_link s
varun-edachali-dbx 99e7435
simplify description extraction
varun-edachali-dbx 54ec080
pass session_id_hex to ThriftResultSet
varun-edachali-dbx f9f9f31
revert to main's extract description
varun-edachali-dbx 51cef2b
validate row count for sync query tests as well
varun-edachali-dbx 387102d
guid_hex -> hex_guid
varun-edachali-dbx d53d1ea
reduce diff
varun-edachali-dbx c7810aa
reduce diff
varun-edachali-dbx b3072bd
reduce diff
varun-edachali-dbx 8be5264
set .value in compression
varun-edachali-dbx 80692e3
reduce diff
varun-edachali-dbx f810ac1
Merge branch 'main' into sea-http-client
varun-edachali-dbx 5a13925
formatting (black)
varun-edachali-dbx 92ba484
remove redundant test
varun-edachali-dbx 2d451b6
move extra_params to the back
varun-edachali-dbx 971969a
Merge branch 'sea-migration' into sea-http-client
varun-edachali-dbx 0e87374
is_direct_results -> has_more_rows
varun-edachali-dbx 20c705f
Revert "is_direct_results -> has_more_rows"
varun-edachali-dbx 2201765
Merge branch 'main' into sea-http-client
varun-edachali-dbx f277b07
stop passing session_id_hex
varun-edachali-dbx e58b6b5
remove redundant comment
varun-edachali-dbx 9e086af
add extra_params param
varun-edachali-dbx 9cfa350
pass extra_params into test_...unset...
varun-edachali-dbx 878a845
remove excess session_id_he
varun-edachali-dbx 071ad4e
reduce changes in DatabricksRetryPolicy
varun-edachali-dbx de339c0
reduce diff in DatabricksRetryPolicy
varun-edachali-dbx 2edff78
simple comments on proxy setting
varun-edachali-dbx 6c5b37f
link docs for getproxies)(
varun-edachali-dbx 328c3bd
Merge branch 'main' into sea-http-client
varun-edachali-dbx dba08cd
rename proxy specific attrs with proxy prefix
varun-edachali-dbx File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.