Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support Spark 3.3.1 version #1183

Merged
merged 1 commit into from
Nov 4, 2022
Merged

Conversation

pawel-big-lebowski
Copy link
Collaborator

@pawel-big-lebowski pawel-big-lebowski commented Oct 27, 2022

Signed-off-by: Pawel Leszczynski [email protected]

Problem

Include support for the latest Spark 3.3.1 version.

Solution

Update Spark version and fix issue introduced by upgraded Jackson within Spark.

Note: All schema changes require discussion. Please link the issue for context.

  • Your change modifies the core OpenLineage model
  • Your change modifies one or more OpenLineage facets

If you're contributing a new integration, please specify the scope of the integration and how/where it has been tested (e.g., Apache Spark integration supports S3 and GCS filesystem operations, tested with AWS EMR).

Checklist

  • You've signed-off your work
  • Your pull request title follows our guidelines
  • Your changes are accompanied by tests (if relevant)
  • Your change contains a small diff and is self-contained
  • You've updated any relevant documentation (if relevant)
  • You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
  • You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
  • You've added a header to source files (if relevant)

@boring-cyborg boring-cyborg bot added area:ci CI area:documentation Improvements or additions to documentation area:integration/spark area:tests Testing code labels Oct 27, 2022
@pawel-big-lebowski
Copy link
Collaborator Author

looks like a Jackson upgrade occurred:
https://issues.apache.org/jira/browse/SPARK-40782

@mobuchowski
Copy link
Member

@pawel-big-lebowski another tree needed? 😉

@pawel-big-lebowski
Copy link
Collaborator Author

@pawel-big-lebowski another tree needed? 😉

Hopefully not. This look like some Spark & Gradle issue that was already merged to Spark master:
FasterXML/jackson-bom#52 (comment)
apache/spark#38355

@pawel-big-lebowski
Copy link
Collaborator Author

Need to wait for docker image bitnami/spark:3.3.1 to proceed.

@pawel-big-lebowski pawel-big-lebowski force-pushed the spark/support-spark-3.3.1 branch 3 times, most recently from e30c403 to 825f50f Compare November 3, 2022 08:53
@pawel-big-lebowski pawel-big-lebowski marked this pull request as ready for review November 3, 2022 09:16
Signed-off-by: Pawel Leszczynski <[email protected]>
@pawel-big-lebowski pawel-big-lebowski merged commit 02f0907 into main Nov 4, 2022
@pawel-big-lebowski pawel-big-lebowski deleted the spark/support-spark-3.3.1 branch November 4, 2022 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:ci CI area:documentation Improvements or additions to documentation area:integration/spark area:tests Testing code
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants