Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tez] Add init action to enable tez-aux-services #1238

Open
cjac opened this issue Sep 25, 2024 · 1 comment
Open

[tez] Add init action to enable tez-aux-services #1238

cjac opened this issue Sep 25, 2024 · 1 comment

Comments

@cjac
Copy link
Contributor

cjac commented Sep 25, 2024

This is an issue requesting to implement tez-aux-services support as an initialization-actions script.

Take a look at the configure_yarn_nodemanager function in spark-rapids.sh[1] for an example of how to set properties in yarn-site.xml. The same code can be used as an example of how to update tez-site.xml as described in the shuffle handler overview[2].

It says here[2] that The Tez Shuffle Handler jar artifact org.apache.org:tez-aux-services needs to be placed into the Node Manager classpath and restarted but does not indicate where to find the jar artifact org.apache.org:tez-aux-services . Thankfully, I was able to look it up pretty quickly. https://mvnrepository.com/artifact/org.apache.tez/tez-aux-services

Be sure to use the following tez-aux-services version based on your dataproc image version.

dataproc 2.0: tez-aux-services 0.9.2 [3]

dataproc 2.1: tez-aux-services 0.10.2 [4]

dataproc 2.2: tez-aux-services 0.10.2 [4]

[1] https://github.com/GoogleCloudDataproc/initialization-actions/blob/master/spark-rapids/spark-rapids.sh#L568

[2] https://tez.apache.org/shuffle-handler.html

[3] https://repo1.maven.org/maven2/org/apache/tez/tez-aux-services/0.9.2/tez-aux-services-0.9.2.jar

[4] https://repo1.maven.org/maven2/org/apache/tez/tez-aux-services/0.10.2/tez-aux-services-0.10.2.jar

@cjac
Copy link
Contributor Author

cjac commented Sep 25, 2024

Any users who would like an easy introduction to contributing to the repository may want to consider this as an introductory exercise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant