-
Notifications
You must be signed in to change notification settings - Fork 28.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-35550][BUILD] Upgrade Jackson to 2.12.3 #32688
Conversation
Kubernetes integration test unable to build dist. exiting with code: 1 |
Test build #139039 has finished for PR 32688 at commit
|
@LuciferYang, the test failure should be now fixed in the latest master branch. |
thx @HyukjinKwon |
Kubernetes integration test unable to build dist. exiting with code: 1 |
Out of curiosity, what is the actual impact for Spark? |
There may not be any real benefits at the moment, if we are going to upgrade avro version in the future, we will need upgrade Jackson to this version because the master of avro already used Jackson 2.12.3. Of course, we can wait until really need to upgrade. |
I see. I would suggest doing the upgrade until we have a better reason for this. |
Test build #139047 has finished for PR 32688 at commit
|
I think it's OK to upgrade - we tend to want Jackson to stay pretty up to date anyway |
@srowen I see. I am OK to merge this as well. |
Merged to master. |
thx all ~ |
Hi all @LuciferYang @HyukjinKwon @gengliangwang @srowen -- While attempting to run a job on k8s with a
The root cause may be that When it initializes but that method has only existed since I will report back if downgrading the pom to 2.11.4 resolves the issue. |
This upgrades from 2.12.2 to 2.12.3 so the compatibility w/ 2.11 is unlikely from this PR. Does that happen after this PR? |
cc @dongjoon-hyun too FYI |
Thank you for pinging me! |
BTW, Apache Spark K8s IT has been healthy. |
@dchristle Could you elaborate about the following a little bit more?
|
@dongjoon-hyun Sure - the error triggers right after the job jar is added, i.e. there are no stages that start, and it's so early that the Spark UI is not accessible. I tried rebuilding my Docker image from scratch and the error persists. I am using the GCS connector and Snowflake connectors for this job, which are technically only on 3.1. I am not sure how to debug whether there is something specific about my build (aka a problem locally) or if there truly is a dependency conflict in the |
Thank you for the detail. Let me build the vanilla Spark distribution from the |
BTW, this is the community Jenkins success result with this PR
|
I am able to build
|
Given the above, could you share your docker image's jar directory, @dchristle ? Do you have both Jackson 2.11.x and 2.12.3? |
Or, is your Spark app using |
That was the root cause. I refactored my workflow to compile Spark from the |
@dongjoon-hyun @HyukjinKwon Thank you for your help, busy with release spark 3.1.x used in our company recently, haha |
What changes were proposed in this pull request?
This pr upgrade Jackson version to 2.12.3.
Jackson Release 2.12.3: https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.12.3
Why are the changes needed?
Upgrade to a new version to bring potential bug fixes like FasterXML/jackson-modules-java8#207 and avro's master has been upgraded to Jackson to 2.12.3
Does this PR introduce any user-facing change?
No
How was this patch tested?
Pass the Jenkins or GitHub Action