Skip to content

Complete Fetch Phase (for INLINE disposition and JSON_ARRAY format) #594

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 57 commits into
base: metadata-sea
Choose a base branch
from
Draft

Complete Fetch Phase (for INLINE disposition and JSON_ARRAY format) #594

wants to merge 57 commits into from

Conversation

varun-edachali-dbx
Copy link
Collaborator

@varun-edachali-dbx varun-edachali-dbx commented Jun 11, 2025 •
edited
Loading

What type of PR is this?

  • Feature

Description

Complete the Fetch phase of the SEA implementation for INLINE disposition and JSON_ARRAY format. Introduce a new JsonQueue.

How is this tested?

  • Unit tests
  • E2E Tests
  • Manually - using the test scripts invoked in examples/experimental/sea_connector_test.py.
  • N/A

The coverage of the key classes by the unit tests are as below:

Module Statements Missing Coverage Notes
utils.py (JsonQueue class) 15 0 100% All methods tested
utils.py (SeaResultSetQueueFactory class) 20 0 100% All methods tested
result_set.py (SeaResultSet class) 60 0 100% All methods tested

Related Tickets & Documents

Design Doc
PECOBLR-484

@varun-edachali-dbx
move guid_to_hex_id import to utils
d97463b 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
reduce diff in guid utils import
139e246 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
move arrow_schema_bytes back into ExecuteResult
e3ee4e4 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
maintain log
f448a8f 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
remove un-necessary assignment
82ca1ee 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
remove un-necessary tuple response
e96a078 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
remove un-ncessary verbose mocking
27158b1 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
move Queue construction to ResultSert
d3200c4 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
move description to List[Tuple]
8a014f0 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
frmatting (black)
39c41ab 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
reduce diff (remove explicit tuple conversion)
2cd04df 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
remove has_more_rows from ExecuteResponse
067a019 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
remove un-necessary has_more_rows aclc
48c83e0 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
default has_more_rows to True
281a9e9 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
return has_more_rows from ExecResponse conversion during GetRespMetadata
192901d 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
remove unnecessary replacement
55f5c45 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
better mocked backend naming
edc36b5 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
remove has_more_rows test in ExecuteResponse
81280e7 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
introduce replacement of original has_more_rows read test
c1d3be2 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
call correct method in test_use_arrow_schema
5ee4136 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
call correct method in test_fall_back_to_hive_schema
b881ab0 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
re-introduce result response read test
53bf715 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
simplify test
45a32be 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
remove excess fetch_results mocks
e3fe299 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
more minimal changes to thrift_backend tests
e8038d3 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
move back to old table types
2f6ec19 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
remove outdated arrow_schema_bytes return
73bc282 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
align SeaResultSet with new structure
4e07f1e 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
correct sea res set tests
65e7c6b 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
minimal fetch phase intro
71b451a 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
remove unimplemented method tests
021ff4c 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
fix sea connector tests
bfc1f01 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx varun-edachali-dbx mentioned this pull request Jun 18, 2025
Draft
5 tasks
@varun-edachali-dbx
remove unused imports
984e8ee 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
remove unused imports
0ce144d 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
run some tests for sea
242307a 
Signed-off-by: varun-edachali-dbx <[email protected]>
@varun-edachali-dbx
pass param as TSparkParameterValue
eb1a9b4 
Signed-off-by: varun-edachali-dbx <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@deeksha-db deeksha-db Awaiting requested review from deeksha-db deeksha-db will be requested when the pull request is marked ready for review deeksha-db is a code owner

@samikshya-db samikshya-db Awaiting requested review from samikshya-db samikshya-db will be requested when the pull request is marked ready for review samikshya-db is a code owner

@jprakash-db jprakash-db Awaiting requested review from jprakash-db jprakash-db will be requested when the pull request is marked ready for review jprakash-db is a code owner

@jackyhu-db jackyhu-db Awaiting requested review from jackyhu-db jackyhu-db will be requested when the pull request is marked ready for review jackyhu-db is a code owner

@madhav-db madhav-db Awaiting requested review from madhav-db madhav-db will be requested when the pull request is marked ready for review madhav-db is a code owner

@gopalldb gopalldb Awaiting requested review from gopalldb gopalldb will be requested when the pull request is marked ready for review gopalldb is a code owner

@jayantsing-db jayantsing-db Awaiting requested review from jayantsing-db jayantsing-db will be requested when the pull request is marked ready for review jayantsing-db is a code owner

@vikrantpuppala vikrantpuppala Awaiting requested review from vikrantpuppala vikrantpuppala will be requested when the pull request is marked ready for review vikrantpuppala is a code owner

@shivam2680 shivam2680 Awaiting requested review from shivam2680 shivam2680 will be requested when the pull request is marked ready for review shivam2680 is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@varun-edachali-dbx