Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Spark 4.0 support #297

Draft
wants to merge 1 commit into
base: develop
Choose a base branch
from
Draft

Add Spark 4.0 support #297

wants to merge 1 commit into from

Conversation

dolfinus
Copy link
Member

@dolfinus dolfinus commented Jul 25, 2024

Change Summary

  • Update CI matrix to include Spark 4.x. By default, it is not used for tests, unless some integration has been changed (like 2.x).
  • Spark 4.x also supports Java 22, include it to Readme & CI matrix.
  • Update JDBCConnection methods to be compatible with Spark 4.0 JDBCUtils methods signatures.
  • Update XML class get_packages() and check_if_supported() methods, as Spark 4.0 includes XML support.
  • Since Spark 4.0, DecimalType(38, 10).typeName() started returning decimal(38, 10) instead of decimal, which breaks some Oracle tests. Updated SparkTypeToHWM implementation to have direct mapping between Spark data type classes and HWM classes.

Notes:

  • Skip Excel tests on Spark 4.0, as there is no package version.

Related issue number

Checklist

  • Commit message and PR title is comprehensive
  • Keep the change as small as possible
  • Unit and integration tests for the changes exist
  • Tests pass on CI and coverage does not decrease
  • Documentation reflects the changes where applicable
  • docs/changelog/next_release/<pull request or issue id>.<change type>.rst file added describing change
    (see CONTRIBUTING.rst for details.)
  • My PR is ready to review.

@dolfinus dolfinus self-assigned this Jul 25, 2024
Copy link

codecov bot commented Jul 25, 2024

Codecov Report

Attention: Patch coverage is 95.83333% with 3 lines in your changes missing coverage. Please review.

Project coverage is 91.97%. Comparing base (1cc7fec) to head (f23637a).

Files with missing lines Patch % Lines
onetl/hwm/store/hwm_class_registry.py 84.61% 1 Missing and 1 partial ⚠️
.../connection/db_connection/jdbc_mixin/connection.py 95.45% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff             @@
##           develop     #297      +/-   ##
===========================================
+ Coverage    91.94%   91.97%   +0.02%     
===========================================
  Files          225      225              
  Lines         9649     9684      +35     
  Branches       987      998      +11     
===========================================
+ Hits          8872     8907      +35     
+ Misses         588      586       -2     
- Partials       189      191       +2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 0e55abd to b91db4e Compare July 25, 2024 18:52
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from b91db4e to 3d1d944 Compare July 25, 2024 19:02
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 3d1d944 to 469c1d9 Compare July 25, 2024 19:08
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 469c1d9 to dd21d0a Compare July 25, 2024 19:20
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from dd21d0a to 776f351 Compare July 25, 2024 19:26
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 776f351 to 0ef84bd Compare July 25, 2024 19:32
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 0ef84bd to d0c9090 Compare July 29, 2024 12:28
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from d0c9090 to 1a00a33 Compare July 29, 2024 12:47
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 1a00a33 to dca6397 Compare July 29, 2024 12:52
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from dca6397 to d050a2d Compare July 29, 2024 13:00
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from d050a2d to 849659e Compare July 29, 2024 13:45
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 849659e to 978dc31 Compare July 29, 2024 13:53
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 978dc31 to e68423d Compare July 29, 2024 14:06
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from e68423d to ffaba47 Compare July 29, 2024 14:11
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from c7d76ba to d97b236 Compare December 3, 2024 13:50
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from d97b236 to fe8208d Compare December 3, 2024 13:58
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from fe8208d to 2b46cbd Compare December 4, 2024 11:00
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 2b46cbd to 0290394 Compare December 4, 2024 11:11
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 0290394 to 2f08db1 Compare December 4, 2024 11:31
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 2f08db1 to 3406360 Compare December 4, 2024 11:45
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 3406360 to bf03e79 Compare December 4, 2024 11:52
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from bf03e79 to 7bb1aac Compare December 4, 2024 13:12
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 7bb1aac to a236028 Compare December 4, 2024 13:54
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from a236028 to d20d9e7 Compare December 4, 2024 14:06
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from d20d9e7 to 068fe01 Compare December 4, 2024 15:24
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from 068fe01 to da0ae6b Compare December 5, 2024 12:02
@dolfinus dolfinus force-pushed the feature/test-spark4.0 branch from da0ae6b to f5ee6f8 Compare December 5, 2024 12:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant