Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NoSuchMethodError exception java.nio.MappedByteBuffer.position in ELF analyzer #2815

Closed
Conan-Kudo opened this issue Jun 18, 2019 · 31 comments
Closed
Labels

Comments

@Conan-Kudo
Copy link

Conan-Kudo commented Jun 18, 2019

I'm trying to run OpenGrok 1.2.14 on Ubuntu 16.04 on Java 8 using the system provided Java 8 and Tomcat 8 stack.

When running the indexer, I got the following tracebacks:

$ sudo opengrok-indexer -J=-Djava.util.logging.config.file=/var/opengrok/logging.properties -a /opt/opengrok/lib/opengrok.jar -- -s /var/opengrok/src -d /var/opengrok/data -H -P -S -G -W /var/opengrok/etc/configuration.xml -c /usr/bin/universal-ctags       
12:59:53 WARNING: Error from ctags: universal-ctags: Warning: Language "clojure" already defined
12:59:53 WARNING: Error from ctags: universal-ctags: Warning: Language "rust" already defined
12:59:53 WARNING: Error from ctags: universal-ctags: Warning: Language "pascal" already defined
12:59:53 WARNING: Error from ctags: universal-ctags: Warning: Language "clojure" already defined
12:59:53 WARNING: Error from ctags: universal-ctags: Warning: Language "rust" already defined
12:59:53 WARNING: Error from ctags: universal-ctags: Warning: Language "pascal" already defined
12:59:53 WARNING: Could not read /var/opengrok/src/device/platform/tests/vendor
13:00:06 WARNING: Error from ctags: universal-ctags: Warning: ignoring null tag in /var/opengrok/src/device/platform/web/UI/js/bootstrap.min.js(line: 6)
13:00:12 SEVERE: 2739 successes (75.3%) after aborting parallel-indexing
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError
        at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006)
        at org.opengrok.indexer.index.IndexDatabase.indexParallel(IndexDatabase.java:1229)
        at org.opengrok.indexer.index.IndexDatabase.update(IndexDatabase.java:496)
        at org.opengrok.indexer.index.IndexDatabase$1.run(IndexDatabase.java:225)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodError
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
        at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)
        ... 8 more
Caused by: java.lang.NoSuchMethodError
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
        at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
        at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735)
        at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:583)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:496)
        at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$2(IndexDatabase.java:1182)
        at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1424)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.lang.NoSuchMethodError: java.nio.MappedByteBuffer.position(I)Ljava/nio/MappedByteBuffer;
        at org.opengrok.indexer.analysis.executables.ELFAnalyzer$ELFHeader.<init>(ELFAnalyzer.java:229)
        at org.opengrok.indexer.analysis.executables.ELFAnalyzer.parseELF(ELFAnalyzer.java:108)
        at org.opengrok.indexer.analysis.executables.ELFAnalyzer.analyze(ELFAnalyzer.java:93)
        at org.opengrok.indexer.analysis.AnalyzerGuru.populateDocument(AnalyzerGuru.java:586)
        at org.opengrok.indexer.index.IndexDatabase.addFile(IndexDatabase.java:754)
        at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$1(IndexDatabase.java:1193)
        at java.util.stream.Collectors.lambda$groupingByConcurrent$51(Collectors.java:1070)
        at java.util.stream.ReferencePipeline.lambda$collect$1(ReferencePipeline.java:496)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
        at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058)
        ... 2 more

13:00:12 WARNING: 3637 failures (100.0%) while parallel-indexing
@vladak vladak changed the title java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError during indexing on OpenGrok 1.2.14 on Java 8 NoSuchMethodError exception java.nio.MappedByteBuffer.position in ELF analyzer Jun 18, 2019
@vladak
Copy link
Member

vladak commented Jun 18, 2019

The code in question in opengrok-indexer/src/main/java/org/opengrok/indexer/analysis/executables/ELFAnalyzer.java:

106      public String parseELF(FileChannel fch) throws IOException {
107          MappedByteBuffer fmap = fch.map(FileChannel.MapMode.READ_ONLY, 0, fch.size());
108          ELFHeader eh = new ELFHeader(fmap);
...
185      private static class ELFHeader {
...
211          ELFHeader(MappedByteBuffer fmap) throws IllegalArgumentException {
212              if (fmap.get(ELFIdentification.EI_MAG0.value()) != 0x7f ||
213                  fmap.get(ELFIdentification.EI_MAG1.value()) != 'E' ||
214                  fmap.get(ELFIdentification.EI_MAG2.value()) != 'L' ||
215                  fmap.get(ELFIdentification.EI_MAG3.value()) != 'F') {
216                  throw new IllegalArgumentException("Not an ELF file");
217              }
218  
219              ei_class = EI_Class.valueOf(fmap.get(ELFIdentification.EI_CLASS.value()));
...
229              fmap.position(ELFIdentification.EI_NIDENT.value());

So, the MappedByteBuffer instance is present (certainly other methods from there were called, otherwise an exception would be thrown earlier), just somehow the position() method is not.

@vladak
Copy link
Member

vladak commented Jun 18, 2019

Actually, what is "system provided Java 8" exactly ?

@Conan-Kudo
Copy link
Author

I'm using the openjdk-8-jdk-headless package from Ubuntu for my Java 8 runtime environment.

@vladak
Copy link
Member

vladak commented Jun 18, 2019

Just to be sure, could you run the indexer with -J=-verbose:class, using the output determine from which file it loads java.nio.Buffer, extract the Buffer.class from that jar file and run javap on it to see if it contains the position() methods ?

@vladak
Copy link
Member

vladak commented Jun 18, 2019

Also, I wonder how many ELF files do you have in your projects and whether all of the suffer from this problem.

@Conan-Kudo
Copy link
Author

Conan-Kudo commented Jun 18, 2019

verbose output:

Jun 18, 2019 3:40:06 PM org.opengrok.indexer.index.IndexDatabase indexParallel
SEVERE: 323 successes (36.0%) after aborting parallel-indexing
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError
        at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006)
        at org.opengrok.indexer.index.IndexDatabase.indexParallel(IndexDatabase.java:1229)
        at org.opengrok.indexer.index.IndexDatabase.update(IndexDatabase.java:496)
        at org.opengrok.indexer.index.IndexDatabase$1.run(IndexDatabase.java:225)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodError
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
        at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)
        ... 8 more
Caused by: java.lang.NoSuchMethodError
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
        at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
        at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735)
        at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:583)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:496)
        at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$2(IndexDatabase.java:1182)
        at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1424)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.lang.NoSuchMethodError: java.nio.MappedByteBuffer.position(I)Ljava/nio/MappedByteBuffer;
        at org.opengrok.indexer.analysis.executables.ELFAnalyzer$ELFHeader.<init>(ELFAnalyzer.java:229)
        at org.opengrok.indexer.analysis.executables.ELFAnalyzer.parseELF(ELFAnalyzer.java:108)
        at org.opengrok.indexer.analysis.executables.ELFAnalyzer.analyze(ELFAnalyzer.java:93)
        at org.opengrok.indexer.analysis.AnalyzerGuru.populateDocument(AnalyzerGuru.java:586)
        at org.opengrok.indexer.index.IndexDatabase.addFile(IndexDatabase.java:754)
        at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$1(IndexDatabase.java:1193)
        at java.util.stream.Collectors.lambda$groupingByConcurrent$51(Collectors.java:1070)
        at java.util.stream.ReferencePipeline.lambda$collect$1(ReferencePipeline.java:496)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
        at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
        ... 4 more

Jun 18, 2019 3:40:06 PM org.opengrok.indexer.index.IndexDatabase indexParallel
WARNING: 898 failures (100.0%) while parallel-indexing

There's only one jar file opengrok.jar, but I popped it open and ran javap on it:

root@opengrok-stg:/opt/opengrok/lib/test/org/opengrok/indexer/analysis/executables# javap ELFAnalyzer.class 
Compiled from "ELFAnalyzer.java"
public class org.opengrok.indexer.analysis.executables.ELFAnalyzer extends org.opengrok.indexer.analysis.FileAnalyzer {
  protected org.opengrok.indexer.analysis.executables.ELFAnalyzer(org.opengrok.indexer.analysis.AnalyzerFactory);
  protected int getSpecializedVersionNo();
  public void analyze(org.apache.lucene.document.Document, org.opengrok.indexer.analysis.StreamSource, java.io.Writer) throws java.io.IOException;
  public java.lang.String parseELF(java.nio.channels.FileChannel) throws java.io.IOException;
  static {};
}

@Conan-Kudo
Copy link
Author

Also, I wonder how many ELF files do you have in your projects and whether all of the suffer from this problem.

ELF files do show up in a fair number of projects...

@vladak
Copy link
Member

vladak commented Jun 18, 2019

For reference on my laptop with Ubuntu 18.04 and Oracle Java 8 (build 1.8.0_181-b13) I stepped through the ELFHeader constructor in debugger (IDEA) past the call to position() without any problem.

@vladak
Copy link
Member

vladak commented Jun 18, 2019

The verbose output should look like this:

$ java -verbose:class -jar ~/opengrok-git-vladak/opengrok-indexer/target/opengrok-1.2.12.jar 2>&1 | head
[Opened /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]
[Loaded java.lang.Object from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]
[Loaded java.io.Serializable from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]
[Loaded java.lang.Comparable from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]
[Loaded java.lang.CharSequence from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]
[Loaded java.lang.String from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]
[Loaded java.lang.reflect.AnnotatedElement from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]
[Loaded java.lang.reflect.GenericDeclaration from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]
[Loaded java.lang.reflect.Type from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]
[Loaded java.lang.Class from /usr/lib/jvm/java-8-oracle/jre/lib/rt.jar]

i.e. the -verbose:class option needs to be passed to java, not as option to the indexer.

@vladak
Copy link
Member

vladak commented Jun 18, 2019

Hence, your indexer line should look like this:

sudo opengrok-indexer -J=-verbose:class -J=-Djava.util.logging.config.file=/var/opengrok/logging.properties -a /opt/opengrok/lib/opengrok.jar -- -s /var/opengrok/src -d /var/opengrok/data -H -P -S -G -W /var/opengrok/etc/configuration.xml -c /usr/bin/universal-ctags

@Conan-Kudo
Copy link
Author

So then the output is this?

[Loaded java.io.RandomAccessFile from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
[Loaded org.opengrok.indexer.analysis.executables.ELFAnalyzer$ELFHeader from file:/opt/opengrok/lib/opengrok.jar]
[Loaded org.opengrok.indexer.analysis.executables.ELFAnalyzer$ELFIdentification from file:/opt/opengrok/lib/opengrok.jar]
[Loaded org.opengrok.indexer.analysis.executables.ELFAnalyzer$EI_Class from file:/opt/opengrok/lib/opengrok.jar]
[Loaded org.opengrok.indexer.analysis.executables.ELFAnalyzer$EI_Data from file:/opt/opengrok/lib/opengrok.jar]
[Loaded java.io.RandomAccessFile$1 from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
[Loaded sun.misc.FormattedFloatingDecimal$Form from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
[Loaded sun.misc.FormattedFloatingDecimal from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
[Loaded sun.misc.FormattedFloatingDecimal$1 from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
[Loaded sun.misc.FormattedFloatingDecimal$2 from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
[Loaded java.io.PrintWriter from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
[Loaded java.lang.Throwable$PrintStreamOrWriter from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
[Loaded java.lang.Throwable$WrappedPrintWriter from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
[Loaded java.util.IdentityHashMap$KeySet from /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar]
16:15:58 SEVERE: 143 successes (40.9%) after aborting parallel-indexing
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError
        at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006)
        at org.opengrok.indexer.index.IndexDatabase.indexParallel(IndexDatabase.java:1229)
        at org.opengrok.indexer.index.IndexDatabase.update(IndexDatabase.java:496)
        at org.opengrok.indexer.index.IndexDatabase$1.run(IndexDatabase.java:225)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodError
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
        at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)
        ... 8 more
Caused by: java.lang.NoSuchMethodError
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
        at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
        at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735)
        at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
        at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:583)
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:496)
        at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$2(IndexDatabase.java:1182)
        at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1424)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.lang.NoSuchMethodError: java.nio.MappedByteBuffer.position(I)Ljava/nio/MappedByteBuffer;
        at org.opengrok.indexer.analysis.executables.ELFAnalyzer$ELFHeader.<init>(ELFAnalyzer.java:229)
        at org.opengrok.indexer.analysis.executables.ELFAnalyzer.parseELF(ELFAnalyzer.java:108)
        at org.opengrok.indexer.analysis.executables.ELFAnalyzer.analyze(ELFAnalyzer.java:93)
        at org.opengrok.indexer.analysis.AnalyzerGuru.populateDocument(AnalyzerGuru.java:586)
        at org.opengrok.indexer.index.IndexDatabase.addFile(IndexDatabase.java:754)
        at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$1(IndexDatabase.java:1193)
        at java.util.stream.Collectors.lambda$groupingByConcurrent$51(Collectors.java:1070)
        at java.util.stream.ReferencePipeline.lambda$collect$1(ReferencePipeline.java:496)
        at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
        at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058)
        ... 2 more

16:15:58 WARNING: 350 failures (100.0%) while parallel-indexing

@Conan-Kudo
Copy link
Author

From the rt.jar mentioned in the log:

root@opengrok-stg:~/foo# javap java/nio/Buffer.class 
Compiled from "Buffer.java"
public abstract class java.nio.Buffer {
  static final int SPLITERATOR_CHARACTERISTICS;
  long address;
  java.nio.Buffer(int, int, int, int);
  public final int capacity();
  public final int position();
  public final java.nio.Buffer position(int);
  public final int limit();
  public final java.nio.Buffer limit(int);
  public final java.nio.Buffer mark();
  public final java.nio.Buffer reset();
  public final java.nio.Buffer clear();
  public final java.nio.Buffer flip();
  public final java.nio.Buffer rewind();
  public final int remaining();
  public final boolean hasRemaining();
  public abstract boolean isReadOnly();
  public abstract boolean hasArray();
  public abstract java.lang.Object array();
  public abstract int arrayOffset();
  public abstract boolean isDirect();
  final int nextGetIndex();
  final int nextGetIndex(int);
  final int nextPutIndex();
  final int nextPutIndex(int);
  final int checkIndex(int);
  final int checkIndex(int, int);
  final int markValue();
  final void truncate();
  final void discardMark();
  static void checkBounds(int, int, int);
}

@Conan-Kudo
Copy link
Author

My OpenJDK version from Ubuntu:

$ dpkg -l | grep openjdk
ii  openjdk-8-jdk-headless:amd64        8u212-b03-0ubuntu1.16.04.1                 amd64        OpenJDK Development Kit (JDK) (headless)
ii  openjdk-8-jre-headless:amd64        8u212-b03-0ubuntu1.16.04.1                 amd64        OpenJDK Java runtime, using Hotspot JIT (headless)

@vladak
Copy link
Member

vladak commented Jun 18, 2019

Could you post the complete output w.r.t. the Loaded lines ?

@Conan-Kudo
Copy link
Author

Attached: opengrok-index-run.txt

@vladak
Copy link
Member

vladak commented Jun 18, 2019

Hm, I don't see anything wrong there.

@Conan-Kudo
Copy link
Author

@vladak Have you tried running 1.2.14 with Ubuntu's OpenJDK?

@vladak
Copy link
Member

vladak commented Jun 19, 2019 via email

@IamTHEvilONE
Copy link
Contributor

Is there any additional data I can provide on this, as I'm seeing something similar? (referenced from #2122)

CentOS Linux release 7.6.1810 (Core)
opengrok-1.2.14
java-1.8.0-openjdk-headless-1.8.0.212.b04-0.el7_6.x86_64
java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64

[Loaded java.nio.MappedByteBuffer from /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.212.b04-0.el7_6.x86_64/jre/lib/rt.jar]

@vladak
Copy link
Member

vladak commented Jun 19, 2019

Maybe you could try to find out if this is data driven, i.e. try to reduce the indexed data to the point when the issue is still happening. I wonder if this happens with just a single ELF file under source root.

@IamTHEvilONE
Copy link
Contributor

So I previously had 10-12 different source codes in my src folder. Cut out all but one to reduce the possible data.

If I ignore any line from std out that's just loading code in verbose mode: this is standard out"

18:17:39 WARNING: Error from ctags: ctags: Warning: Language "clojure" already defined
18:17:39 WARNING: Error from ctags: ctags: Warning: Language "rust" already defined
18:17:39 WARNING: Error from ctags: ctags: Warning: Language "pascal" already defined
18:17:39 WARNING: Error from ctags: ctags: Warning: Language "clojure" already defined
18:17:39 WARNING: Error from ctags: ctags: Warning: Language "rust" already defined
18:17:39 WARNING: Error from ctags: ctags: Warning: Language "pascal" already defined
18:17:39 WARNING: Error from ctags: ctags: Warning: Language "clojure" already defined
18:17:39 WARNING: Error from ctags: ctags: Warning: Language "rust" already defined
18:17:39 WARNING: Error from ctags: ctags: Warning: Language "pascal" already defined
18:17:39 WARNING: Error from ctags: ctags: Warning: Language "clojure" already defined
18:17:39 WARNING: Error from ctags: ctags: Warning: Language "rust" already defined
18:17:39 WARNING: Error from ctags: ctags: Warning: Language "pascal" already defined
18:17:46 SEVERE: 1813 successes (40.4%) after aborting parallel-indexing
java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError
at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1006)
at org.opengrok.indexer.index.IndexDatabase.indexParallel(IndexDatabase.java:1229)
at org.opengrok.indexer.index.IndexDatabase.update(IndexDatabase.java:496)
at org.opengrok.indexer.index.IndexDatabase$1.run(IndexDatabase.java:225)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoSuchMethodError
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
at java.util.concurrent.ForkJoinTask.get(ForkJoinTask.java:1005)
... 8 more
Caused by: java.lang.NoSuchMethodError
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at java.util.concurrent.ForkJoinTask.getThrowableException(ForkJoinTask.java:598)
at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:677)
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:735)
at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418)
at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:583)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:496)
at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$2(IndexDatabase.java:1182)
at java.util.concurrent.ForkJoinTask$AdaptedCallable.exec(ForkJoinTask.java:1424)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Caused by: java.lang.NoSuchMethodError: java.nio.MappedByteBuffer.position(I)Ljava/nio/MappedByteBuffer;
at org.opengrok.indexer.analysis.executables.ELFAnalyzer$ELFHeader.(ELFAnalyzer.java:229)
at org.opengrok.indexer.analysis.executables.ELFAnalyzer.parseELF(ELFAnalyzer.java:108)
at org.opengrok.indexer.analysis.executables.ELFAnalyzer.analyze(ELFAnalyzer.java:93)
at org.opengrok.indexer.analysis.AnalyzerGuru.populateDocument(AnalyzerGuru.java:586)
at org.opengrok.indexer.index.IndexDatabase.addFile(IndexDatabase.java:754)
at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$1(IndexDatabase.java:1193)
at java.util.stream.Collectors.lambda$groupingByConcurrent$51(Collectors.java:1070)
at java.util.stream.ReferencePipeline.lambda$collect$1(ReferencePipeline.java:496)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058)
... 2 more
18:17:46 WARNING: 4487 failures (100.0%) while parallel-indexing

18:17:52 SEVERE: ERROR addFile():
org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:680)
at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:694)
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1613)
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1235)
at org.opengrok.indexer.index.IndexDatabase.addFile(IndexDatabase.java:780)
at org.opengrok.indexer.index.IndexDatabase.lambda$indexParallel$1(IndexDatabase.java:1193)
at java.util.stream.Collectors.lambda$groupingByConcurrent$51(Collectors.java:1070)
at java.util.stream.ReferencePipeline.lambda$collect$1(ReferencePipeline.java:496)
at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)

18:17:52 WARNING: An error occurred while finishing writer and completer

18:17:52 WARNING: An error occurred while finishing writer and completer
java.util.ConcurrentModificationException
at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1561)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
at java.util.stream.AbstractTask.compute(AbstractTask.java:316)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at org.opengrok.indexer.index.PendingFileCompleter.completeRenamings(PendingFileCompleter.java:195)
at org.opengrok.indexer.index.PendingFileCompleter.complete(PendingFileCompleter.java:171)
at org.opengrok.indexer.index.IndexDatabase.finishWriting(IndexDatabase.java:1660)
at org.opengrok.indexer.index.IndexDatabase.update(IndexDatabase.java:519)
at org.opengrok.indexer.index.IndexDatabase$1.run(IndexDatabase.java:225)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

18:17:52 SEVERE: Failed with unexpected RuntimeException
java.util.ConcurrentModificationException
at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1561)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
at java.util.stream.AbstractTask.compute(AbstractTask.java:316)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at org.opengrok.indexer.index.PendingFileCompleter.completeRenamings(PendingFileCompleter.java:195)
at org.opengrok.indexer.index.PendingFileCompleter.complete(PendingFileCompleter.java:171)
at org.opengrok.indexer.index.IndexDatabase.finishWriting(IndexDatabase.java:1660)
at org.opengrok.indexer.index.IndexDatabase.update(IndexDatabase.java:519)
at org.opengrok.indexer.index.IndexDatabase$1.run(IndexDatabase.java:225)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

18:17:52 SEVERE: Problem updating index database in directory /usr/opengrok/data-2/index/index-test:
java.util.ConcurrentModificationException
at java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1561)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747)
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721)
at java.util.stream.AbstractTask.compute(AbstractTask.java:316)
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinTask.doInvoke(ForkJoinTask.java:401)
at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:734)
at java.util.stream.ReduceOps$ReduceOp.evaluateParallel(ReduceOps.java:714)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at org.opengrok.indexer.index.PendingFileCompleter.completeRenamings(PendingFileCompleter.java:195)
at org.opengrok.indexer.index.PendingFileCompleter.complete(PendingFileCompleter.java:171)
at org.opengrok.indexer.index.IndexDatabase.finishWriting(IndexDatabase.java:1660)
at org.opengrok.indexer.index.IndexDatabase.update(IndexDatabase.java:519)
at org.opengrok.indexer.index.IndexDatabase$1.run(IndexDatabase.java:225)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

I'll see if I can scrub and attach the opengrok0.0.log

@carbonrobin
Copy link

I'm experiencing this problem also. Searching around shows other applications experiencing the problem on Java 8 when built with Java 9 or greater. I'm new to Opengrok, but did I see the releases are built using Java 11? Hopefully what I found helps.

Source: [https://github.com/lmdbjava/lmdbjava/issues/116]

Java 9 introduces overridden methods with covariant return types for the following methods in java.nio.ByteBuffer that are used by the driver

In Java 9 they all now return ByteBuffer, whereas the methods they override return Buffer,
resulting in exceptions like this when executing on Java 8 and lower

This is because the generated byte code includes the static return type of the method, which is not found on Java 8 and lower because the overloaded methods with covariant return types don't exist.

The solution is to cast ByteBuffer instances to Buffer before calling the method.

@carbonrobin
Copy link

And it looks like a better option would be to add the flag --release 8 to javac when building.

Source: plasma-umass/doppio#497
Source: jetty/jetty.project#3244
Source: https://www.reddit.com/r/androiddev/comments/ah194v/javaniobytebuffer_java_9_compiler_and_android/

@vladak
Copy link
Member

vladak commented Jun 20, 2019

This might be it. The deploy stage in Travis basically builds the whole distribution from scratch (which is another problem of its own) using JDK 11 (e.g. https://travis-ci.org/oracle/opengrok/jobs/548094045 for 1.2.15).

Curiously, Maven's has the compileTarget property set to 1.8.

@vladak
Copy link
Member

vladak commented Jun 20, 2019

I changed the deploy build in cset 54bbf93, will trigger new release.

@vladak
Copy link
Member

vladak commented Jun 20, 2019

relevant comment on the setting the compile target: plasma-umass/doppio#497 (comment)

@vladak
Copy link
Member

vladak commented Jun 20, 2019

It seems the ultimate cure for this problem is to use --release 8 as explained on jetty/jetty.project#3244 (comment) , however as I understand it one needs JDK 9+ for that.

@vladak
Copy link
Member

vladak commented Jun 20, 2019

@Conan-Kudo , @IamTHEvilONE, @carbonrobin once 1.2.16 is out please give it a try.

@IamTHEvilONE
Copy link
Contributor

I have no parallel-indexing errors on 1.2.16 on a single source code repo, nor MappedByteBuffer errors.

Testing with my 12+ code bases I'd index.

@Conan-Kudo
Copy link
Author

It seems to be okay for me so far...

@vladak
Copy link
Member

vladak commented Jun 21, 2019

Cool, closing then. Many thanks @carbonrobin !

@vladak vladak closed this as completed Jun 21, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants