-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run into java.lang.OutOfMemoryError problem after upgraded to InterProScan-5.72-103.0 #392
Comments
Hi @irislin886, How many sequences are in your input file? The error seems to indicate that interProScan ran out of memory while holding too much data in memory, and this is often associated with large input files. |
Hi @matthiasblum , Thank you very much for getting back to me. I have 16 input files that I want to run the InterProScan on. The number of sequences in each file varies from 1501 to 8136. Should I split the input files so each input file contains fewer sequences? |
These numbers sounds reasonable. Are you running InterProScan on a personal device (laptop, desktop), or on a computer farm? How much memory is available? |
I am running InterProScan on my school cluster. I used 1 node, 3 cpu and 20GB per cpu. |
Then I would increase the memory (to something like 40GB) and try again. |
I tried to increase the memory but now it showed another error. I removed the interproscan-5.72-103.0 and downloaded it again. This time, I got an error when I tried to run the test command "./interproscan.sh -i test_all_appl.fasta -f tsv -dp". Here is the error message: 2024-12-23 12:16:41,185 [amqEmbeddedWorkerJmsContainer-7] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:216] ERROR - Execution thrown when attempting to executeInTransaction the StepExecution. All database activity rolled back.
2024-12-23 12:16:41,194 [amqEmbeddedWorkerJmsContainer-7] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:218] ERROR - The exception is :
2024-12-23 12:16:41,195 [amqEmbeddedWorkerJmsContainer-7] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:222] ERROR - StepExecution with errors - stepName: stepPrositePatternRunBinary 2024-12-23 12:16:43,884 [amqEmbeddedWorkerJmsContainer-4] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:216] ERROR - Execution thrown when attempting to executeInTransaction the StepExecution. All database activity rolled back.
2024-12-23 12:16:43,885 [amqEmbeddedWorkerJmsContainer-4] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:218] ERROR - The exception is :
2024-12-23 12:16:43,886 [amqEmbeddedWorkerJmsContainer-4] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:222] ERROR - StepExecution with errors - stepName: stepPrositeProfilesRunBinary 2024-12-23 12:16:54,792 [amqEmbeddedWorkerJmsContainer-8] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:216] ERROR - Execution thrown when attempting to executeInTransaction the StepExecution. All database activity rolled back.
2024-12-23 12:16:54,793 [amqEmbeddedWorkerJmsContainer-8] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:218] ERROR - The exception is :
2024-12-23 12:16:54,794 [amqEmbeddedWorkerJmsContainer-8] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:222] ERROR - StepExecution with errors - stepName: stepPrositePatternRunBinary 2024-12-23 12:16:56,410 [amqEmbeddedWorkerJmsContainer-2] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:216] ERROR - Execution thrown when attempting to executeInTransaction the StepExecution. All database activity rolled back.
2024-12-23 12:16:56,411 [amqEmbeddedWorkerJmsContainer-2] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:218] ERROR - The exception is :
2024-12-23 12:16:56,411 [amqEmbeddedWorkerJmsContainer-2] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:222] ERROR - StepExecution with errors - stepName: stepPrositeProfilesRunBinary 2024-12-23 12:17:00,285 [amqEmbeddedWorkerJmsContainer-6] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:216] ERROR - Execution thrown when attempting to executeInTransaction the StepExecution. All database activity rolled back.
2024-12-23 12:17:00,285 [amqEmbeddedWorkerJmsContainer-6] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:218] ERROR - The exception is :
2024-12-23 12:17:00,286 [amqEmbeddedWorkerJmsContainer-6] [uk.ac.ebi.interpro.scan.jms.worker.LocalJobQueueListener:222] ERROR - StepExecution with errors - stepName: stepPrositePatternRunBinary |
Hi @irislin886 The cause of the error can be found in the logs you posted:
The PROSITE binary requires GLIBC 2.27 or higher, which you don't seem to have. You should contact the system administrator of your cluster and ask them to update GLIBC. Alternatively, you can use InterProScan with a container runtime such as Docker or Singularity: https://interproscan-docs.readthedocs.io/en/latest/HowToUseViaContainer.html |
Hi, I have been using the 5.71-102.0-64-bit version of interproscan on my fasta files and I did not have anyproblem running it. However, when I tried to use it again, it showed:
"The version of InterProScan you are using is 5.71-102.0
The version of the lookup service you are using is 5.72-103.0
As the data in these versions is not the same, you cannot use this match lookup service.
InterProScan will now run locally"
So I followed the instruction a get the newest version which is 5.72-103.0 and reran it. However, I got the error message like this:
02/12/2024 14:53:34:011 Welcome to InterProScan-5.72-103.0
02/12/2024 14:53:34:012 Running InterProScan v5 in STANDALONE mode... on Linux
02/12/2024 14:53:39:008 RunID: tc123_20241202_145338288_bz6a
02/12/2024 14:53:48:846 Loading file /globalscratch/iris_job/46_annotated_Ref_Drosophila/5_InterProScan_prep/update/ananassae_significant_genes.fasta
02/12/2024 14:53:48:848 Running the following analyses:
[AntiFam-7.0,CDD-3.20,Coils-2.2.1,FunFam-4.3.0,Gene3D-4.3.0,Hamap-2023_05,MobiDBLite-4.0,NCBIfam-15.0,PANTHER-19.0,Pfam-37.1,PIRSF-3.10,PIRSR-2023_05,PRINTS-42.0,ProSitePatterns-2023_05,ProSiteProfiles-2023_05,SFLD-4,SMART-9.0,SUPERFAMILY-1.75]
Available matches will be retrieved from the pre-calculated match lookup service.
Matches for any sequences that are not represented in the lookup service will be calculated locally.
org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException: java.lang.OutOfMemoryError: Java heap space
at org.iq80.leveldb.impl.DbImpl.checkBackgroundException(DbImpl.java:421)
at org.iq80.leveldb.impl.DbImpl.writeInternal(DbImpl.java:683)
at org.iq80.leveldb.impl.DbImpl.put(DbImpl.java:649)
at org.iq80.leveldb.impl.DbImpl.put(DbImpl.java:642)
at uk.ac.ebi.interpro.scan.persistence.kvstore.LevelDBStore.put(LevelDBStore.java:78)
at uk.ac.ebi.interpro.scan.persistence.kvstore.LevelDBStore.put(LevelDBStore.java:64)
at uk.ac.ebi.interpro.scan.persistence.MatchDAOImpl.persist(MatchDAOImpl.java:49)
at jdk.internal.reflect.GeneratedMethodAccessor88.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:205)
at jdk.proxy2/jdk.proxy2.$Proxy80.persist(Unknown Source)
at uk.ac.ebi.interpro.scan.precalc.berkeley.conversion.toi5.LookupStoreToI5ModelDAOImpl.persistMatch(LookupStoreToI5ModelDAOImpl.java:441)
at uk.ac.ebi.interpro.scan.precalc.berkeley.conversion.toi5.LookupStoreToI5ModelDAOImpl.populateProteinMatches(LookupStoreToI5ModelDAOImpl.java:340)
at jdk.internal.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at jdk.proxy2/jdk.proxy2.$Proxy104.populateProteinMatches(Unknown Source)
at uk.ac.ebi.interpro.scan.business.sequence.BerkeleyPrecalculatedProteinLookup.getPrecalculated(BerkeleyPrecalculatedProteinLookup.java:488)
at uk.ac.ebi.interpro.scan.business.sequence.BerkeleyPrecalculatedProteinLookup.run(BerkeleyPrecalculatedProteinLookup.java:158)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.OutOfMemoryError: Java heap space
java.lang.IllegalStateException: Problem inserting data into the DBStore org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException: java.lang.OutOfMemoryError: Java heap space
at uk.ac.ebi.interpro.scan.persistence.kvstore.LevelDBStore.put(LevelDBStore.java:91)
at uk.ac.ebi.interpro.scan.persistence.kvstore.LevelDBStore.put(LevelDBStore.java:64)
at uk.ac.ebi.interpro.scan.persistence.MatchDAOImpl.persist(MatchDAOImpl.java:49)
at jdk.internal.reflect.GeneratedMethodAccessor88.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:205)
at jdk.proxy2/jdk.proxy2.$Proxy80.persist(Unknown Source)
at uk.ac.ebi.interpro.scan.precalc.berkeley.conversion.toi5.LookupStoreToI5ModelDAOImpl.persistMatch(LookupStoreToI5ModelDAOImpl.java:441)
at uk.ac.ebi.interpro.scan.precalc.berkeley.conversion.toi5.LookupStoreToI5ModelDAOImpl.populateProteinMatches(LookupStoreToI5ModelDAOImpl.java:340)
at jdk.internal.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at jdk.proxy2/jdk.proxy2.$Proxy104.populateProteinMatches(Unknown Source)
at uk.ac.ebi.interpro.scan.business.sequence.BerkeleyPrecalculatedProteinLookup.getPrecalculated(BerkeleyPrecalculatedProteinLookup.java:488)
at uk.ac.ebi.interpro.scan.business.sequence.BerkeleyPrecalculatedProteinLookup.run(BerkeleyPrecalculatedProteinLookup.java:158)
at java.base/java.lang.Thread.run(Thread.java:833)
org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException: java.lang.OutOfMemoryError: Java heap space
at org.iq80.leveldb.impl.DbImpl.checkBackgroundException(DbImpl.java:421)
at org.iq80.leveldb.impl.DbImpl.writeInternal(DbImpl.java:683)
at org.iq80.leveldb.impl.DbImpl.put(DbImpl.java:649)
at org.iq80.leveldb.impl.DbImpl.put(DbImpl.java:642)
at uk.ac.ebi.interpro.scan.persistence.kvstore.LevelDBStore.put(LevelDBStore.java:78)
at uk.ac.ebi.interpro.scan.persistence.kvstore.LevelDBStore.put(LevelDBStore.java:64)
at uk.ac.ebi.interpro.scan.persistence.MatchDAOImpl.persist(MatchDAOImpl.java:49)
at jdk.internal.reflect.GeneratedMethodAccessor88.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:205)
at jdk.proxy2/jdk.proxy2.$Proxy80.persist(Unknown Source)
at uk.ac.ebi.interpro.scan.precalc.berkeley.conversion.toi5.LookupStoreToI5ModelDAOImpl.persistMatch(LookupStoreToI5ModelDAOImpl.java:441)
at uk.ac.ebi.interpro.scan.precalc.berkeley.conversion.toi5.LookupStoreToI5ModelDAOImpl.populateProteinMatches(LookupStoreToI5ModelDAOImpl.java:364)
at jdk.internal.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at jdk.proxy2/jdk.proxy2.$Proxy104.populateProteinMatches(Unknown Source)
at uk.ac.ebi.interpro.scan.business.sequence.BerkeleyPrecalculatedProteinLookup.getPrecalculated(BerkeleyPrecalculatedProteinLookup.java:488)
at uk.ac.ebi.interpro.scan.business.sequence.BerkeleyPrecalculatedProteinLookup.run(BerkeleyPrecalculatedProteinLookup.java:158)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.OutOfMemoryError: Java heap space
java.lang.IllegalStateException: Problem inserting data into the DBStore org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException: java.lang.OutOfMemoryError: Java heap space
at uk.ac.ebi.interpro.scan.persistence.kvstore.LevelDBStore.put(LevelDBStore.java:91)
at uk.ac.ebi.interpro.scan.persistence.kvstore.LevelDBStore.put(LevelDBStore.java:64)
at uk.ac.ebi.interpro.scan.persistence.MatchDAOImpl.persist(MatchDAOImpl.java:49)
at jdk.internal.reflect.GeneratedMethodAccessor88.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:205)
at jdk.proxy2/jdk.proxy2.$Proxy80.persist(Unknown Source)
at uk.ac.ebi.interpro.scan.precalc.berkeley.conversion.toi5.LookupStoreToI5ModelDAOImpl.persistMatch(LookupStoreToI5ModelDAOImpl.java:441)
at uk.ac.ebi.interpro.scan.precalc.berkeley.conversion.toi5.LookupStoreToI5ModelDAOImpl.populateProteinMatches(LookupStoreToI5ModelDAOImpl.java:364)
at jdk.internal.reflect.GeneratedMethodAccessor91.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at jdk.proxy2/jdk.proxy2.$Proxy104.populateProteinMatches(Unknown Source)
at uk.ac.ebi.interpro.scan.business.sequence.BerkeleyPrecalculatedProteinLookup.getPrecalculated(BerkeleyPrecalculatedProteinLookup.java:488)
at uk.ac.ebi.interpro.scan.business.sequence.BerkeleyPrecalculatedProteinLookup.run(BerkeleyPrecalculatedProteinLookup.java:158)
at java.base/java.lang.Thread.run(Thread.java:833)
org.iq80.leveldb.impl.DbImpl$BackgroundProcessingException: java.lang.OutOfMemoryError: Java heap space
at org.iq80.leveldb.impl.DbImpl.checkBackgroundException(DbImpl.java:421)
at org.iq80.leveldb.impl.DbImpl.get(DbImpl.java:584) ...
Does anyone know what is going on and how can I fix it? Is it possible to use the old version instead? Thank you very much.
The text was updated successfully, but these errors were encountered: