java.lang.NoSuchMethodError: okio.ByteString.encodeString with Spark/InfluxDB

[yeenow123]

When using the InfluxDB reporter on Spark 2.2.0 with Ambari HDP, I'm getting the following error.

I ran mvn -P influxdb clean package to build the jar.

Using the following command (with classname / host replaced):

spark-submit --master yarn-cluster --class com.output.spark.FilteringJob --conf spark.jars=hdfs:///user/smorgasborg/lib/jvm-profiler-1.0.0.jar --conf spark.executor.extraJavaOptions=-javaagent:jvm-profiler-1.0.0.jar=reporter=com.uber.profiling.reporters.InfluxDBOutputReporter,tag=profiling,sampleInterval=1000,influxdb.host={{HOST HERE}},influxdb.port=8086,influxdb.database=test_profiling --conf spark.yarn.am.waitTime=200s spark-output-assembly.jar

[WARNING] 1544111654753 com.uber.profiling.ProfilerRunner: Failed to run profile: com.uber.profiling.profilers.StacktraceReporterProfiler@b9afc07 java.lang.NoSuchMethodError: okio.ByteString.encodeString(Ljava/lang/String;Ljava/nio/charset/Charset;)Lokio/ByteString;
	at okhttp3.Credentials.basic(Credentials.java:35)
	at okhttp3.Credentials.basic(Credentials.java:30)
	at ujagent_shaded.org.influxdb.impl.BasicAuthInterceptor.<init>(BasicAuthInterceptor.java:15)
	at ujagent_shaded.org.influxdb.impl.InfluxDBImpl.<init>(InfluxDBImpl.java:153)
	at ujagent_shaded.org.influxdb.impl.InfluxDBImpl.<init>(InfluxDBImpl.java:122)
	at ujagent_shaded.org.influxdb.impl.InfluxDBImpl.<init>(InfluxDBImpl.java:185)
	at ujagent_shaded.org.influxdb.InfluxDBFactory.connect(InfluxDBFactory.java:48)
	at com.uber.profiling.reporters.InfluxDBOutputReporter.ensureInfluxDBCon(InfluxDBOutputReporter.java:148)
	at com.uber.profiling.reporters.InfluxDBOutputReporter.report(InfluxDBOutputReporter.java:57)
	at com.uber.profiling.profilers.StacktraceReporterProfiler.profile(StacktraceReporterProfiler.java:118)
	at com.uber.profiling.ProfilerRunner.run(ProfilerRunner.java:38)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

[yeenow123]

Created this pull request, okio version in the influxdb dependency conflicted with the Spark okio version

#31

[baghelamit]

Hi @yeenow123, I couldn't reproduce this error with Spark-2.2.0. I checked the spark-deps-hadoop-2.7 for spark branch-2.2 but this branch doesn't show any dependency for okhttp and okio jar files. branch-2.3 and branch-2.4 listed the dependency for okhttp and okio jar files and versions given are compatible with InfluxDBOutputReporter. It looks like there is some dependency in your spark application jar which is bringing the older version of okhttp and okio jar files. Could you please check the dependency tree for your spark application and identify that jar file? Please also check the version of okhttp and okio jar files in that dependency tree. This will help us to narrow down the issue and applying the fix. Thanks.

[c-horn]

I am also seeing this error with spark 2.3

I am bringing in 1.14 okio via --repositories:

[info]   | | +-org.influxdb:influxdb-java:2.14
[info]   | |   +-com.squareup.okhttp3:logging-interceptor:3.11.0
[info]   | |   | +-com.squareup.okhttp3:okhttp:3.10.0 (evicted by: 3.11.0)
[info]   | |   | +-com.squareup.okhttp3:okhttp:3.11.0
[info]   | |   |   +-com.squareup.okio:okio:1.14.0
[info]   | |   |
[info]   | |   +-com.squareup.okhttp3:okhttp:3.10.0 (evicted by: 3.11.0)
[info]   | |   +-com.squareup.okhttp3:okhttp:3.11.0
[info]   | |   | +-com.squareup.okio:okio:1.14.0
[info]   | |   |
[info]   | |   +-com.squareup.retrofit2:converter-moshi:2.4.0
[info]   | |   | +-com.squareup.moshi:moshi:1.5.0
[info]   | |   | | +-com.squareup.okio:okio:1.13.0 (evicted by: 1.14.0)
[info]   | |   | | +-com.squareup.okio:okio:1.14.0
[info]   | |   | |
[info]   | |   | +-com.squareup.retrofit2:retrofit:2.4.0
[info]   | |   |   +-com.squareup.okhttp3:okhttp:3.10.0 (evicted by: 3.11.0)
[info]   | |   |   +-com.squareup.okhttp3:okhttp:3.11.0
[info]   | |   |     +-com.squareup.okio:okio:1.14.0
[info]   | |   |
[info]   | |   +-com.squareup.retrofit2:retrofit:2.4.0
[info]   | |   | +-com.squareup.okhttp3:okhttp:3.10.0 (evicted by: 3.11.0)
[info]   | |   | +-com.squareup.okhttp3:okhttp:3.11.0
[info]   | |   |   +-com.squareup.okio:okio:1.14.0
[info]   | |   |
[info]   | |   +-org.msgpack:msgpack-core:0.8.16

Nothing else on my dependency list pulls in any square/okio/okhttp.

[c-horn]

It appears this only happens in cluster mode, with client mode I do not get the error running an identical spark job.

The cluster mode submit is including this jar on the system classpath:
/usr/hdp/current/hadoop-hdfs-client/lib/okio-1.4.0.jar

EDIT: correction, in client mode it appears the driver does not receive the jar in question, but the executors still do.

[felixcheung]

in client mode, your driver is the spark submit process...

it looks like /usr/hdp/current/hadoop-hdfs-client/lib/okio-1.4.0.jar is injected by YARN?

[g1thubhub]

Hey @felixcheung and @yeenow123, this issue might also be caused by the relocations in the POM file -- I had a similar "class not found issue" when testing a new OutputReporter that I'll commit soon, see this PR for how I fixed it the issue:
#51

[hanhtd2]

Actually in my case, I fixed it.
I used maven and change the version of influxdb:
Try this:

org.influxdb
influxdb-java
2.7

[hiboyang]

Thanks hanhtd2 for the information! I updated influxdb-java to 2.7 in pom.xml just now.