Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle extra columns present on Spanner #1995

Merged
merged 6 commits into from
Nov 7, 2024

Conversation

Deep1998
Copy link
Contributor

@Deep1998 Deep1998 commented Nov 7, 2024

Description

This PR supports Spanner having extra columns not present at source. Earlier, an additional column was a TODO, so apart from the shard id, extra cols would cause the transformation to error out. Additional columns on Spanner can be populated via

  1. Custom transformations
  2. Autogeneration like generated columns, default columns etc. (These are now omitted from the mutation)
  3. Synthetic PK (These are populated with a UUID like the live template)
  4. ShardId (This was already handled previously)

Testing

Unit Tests Added

Pending

  • Manual migration using extra columns on Spanner.

@Deep1998 Deep1998 requested a review from a team as a code owner November 7, 2024 08:20
@Deep1998 Deep1998 requested review from VardhanThigle and removed request for manitgupta November 7, 2024 08:20
@pull-request-size pull-request-size bot added size/XL and removed size/L labels Nov 7, 2024
Copy link

codecov bot commented Nov 7, 2024

Codecov Report

Attention: Patch coverage is 82.22222% with 8 lines in your changes missing coverage. Please review.

Project coverage is 52.86%. Comparing base (15609c3) to head (02818d3).
Report is 2 commits behind head on main.

Files with missing lines Patch % Lines
.../spanner/migrations/schema/SessionBasedMapper.java 78.57% 6 Missing ⚠️
...er/migrations/avro/GenericRecordTypeConvertor.java 86.66% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #1995      +/-   ##
============================================
+ Coverage     45.26%   52.86%   +7.59%     
+ Complexity     3632     1345    -2287     
============================================
  Files           836      372     -464     
  Lines         49872    20616   -29256     
  Branches       5253     2088    -3165     
============================================
- Hits          22577    10898   -11679     
+ Misses        25621     9041   -16580     
+ Partials       1674      677     -997     
Components Coverage Δ
spanner-templates 67.86% <82.22%> (+1.23%) ⬆️
spanner-import-export ∅ <ø> (∅)
spanner-live-forward-migration 75.93% <82.22%> (+0.06%) ⬆️
spanner-live-reverse-replication 76.69% <82.22%> (+0.05%) ⬆️
spanner-bulk-migration 86.38% <82.22%> (-0.04%) ⬇️
Files with missing lines Coverage Δ
...t/v2/spanner/migrations/schema/IdentityMapper.java 100.00% <100.00%> (ø)
...er/migrations/avro/GenericRecordTypeConvertor.java 87.24% <86.66%> (-0.10%) ⬇️
.../spanner/migrations/schema/SessionBasedMapper.java 95.62% <78.57%> (-4.38%) ⬇️

... and 481 files with indirect coverage changes

Copy link
Contributor

@bharadwaj-aditya bharadwaj-aditya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also check if this works for the tokenList use case ?

VardhanThigle
VardhanThigle previously approved these changes Nov 7, 2024
@Deep1998
Copy link
Contributor Author

Deep1998 commented Nov 7, 2024

TOKENLIST is a new datatype in Spanner that is not handled here. That will be taken up in a follow up PR.

Copy link
Contributor

@bharadwaj-aditya bharadwaj-aditya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@Deep1998 Deep1998 merged commit a4bf9d4 into GoogleCloudPlatform:main Nov 7, 2024
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants