Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

求助 Failed to run job : Application application_1591113812497_103797 failed 2 times 作业运行失败 #65

Open
boylige opened this issue Jun 25, 2020 · 1 comment

Comments

@boylige
Copy link

boylige commented Jun 25, 2020

[2020-06-25 22:33:23.064]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=100M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=200M; support was removed in 8.0

[2020-06-25 22:33:23.065]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=100M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=200M; support was removed in 8.0

For more detailed output, check the application tracking page: http://ecs-hn1b-bd-cdp-edg-2:8188/applicationhistory/app/application_1591113812497_103797 Then click on links to logs of each attempt.
. Failing the application.
at com.tencent.angel.client.yarn.AngelYarnClient.updateMaster(AngelYarnClient.java:517)
at com.tencent.angel.client.yarn.AngelYarnClient.startPSServer(AngelYarnClient.java:170)
at com.tencent.angel.client.AngelPSClient.startPS(AngelPSClient.java:115)
at com.tencent.angel.sona.core.DriverContext.startAngelAndPSAgent(DriverContext.scala:97)
at com.tencent.angel.sona.examples.JsonRunnerExamples$.main(JsonRunnerExamples.scala:69)
at com.tencent.angel.sona.examples.JsonRunnerExamples.main(JsonRunnerExamples.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Exception in thread "main" com.tencent.angel.exception.AngelException: java.io.IOException: Failed to run job : Application application_1591113812497_103797 failed 2times (global limit =3; local limit is =2) due to AM Container for appattempt_1591113812497_103797_000002 exited with exitCode: 1
Failing this attempt.Diagnostics: [2020-06-25 22:33:23.063]Exception from container-launch.
Container id: container_e74_1591113812497_103797_02_000001
Exit code: 1

[2020-06-25 22:33:23.064]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=100M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=200M; support was removed in 8.0

[2020-06-25 22:33:23.065]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=100M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=200M; support was removed in 8.0

For more detailed output, check the application tracking page: http://ecs-hn1b-bd-cdp-edg-2:8188/applicationhistory/app/application_1591113812497_103797 Then click on links to logs of each attempt.
. Failing the application.
at com.tencent.angel.client.yarn.AngelYarnClient.startPSServer(AngelYarnClient.java:176)
at com.tencent.angel.client.AngelPSClient.startPS(AngelPSClient.java:115)
at com.tencent.angel.sona.core.DriverContext.startAngelAndPSAgent(DriverContext.scala:97)
at com.tencent.angel.sona.examples.JsonRunnerExamples$.main(JsonRunnerExamples.scala:69)
at com.tencent.angel.sona.examples.JsonRunnerExamples.main(JsonRunnerExamples.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:851)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:926)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:935)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.io.IOException: Failed to run job : Application application_1591113812497_103797 failed 2 times (global limit =3; local limit is =2) due to AM Container for appattempt_1591113812497_103797_000002 exited with exitCode: 1
Failing this attempt.Diagnostics: [2020-06-25 22:33:23.063]Exception from container-launch.
Container id: container_e74_1591113812497_103797_02_000001
Exit code: 1

[2020-06-25 22:33:23.064]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=100M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=200M; support was removed in 8.0

[2020-06-25 22:33:23.065]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=100M; support was removed in 8.0
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=200M; support was removed in 8.0

For more detailed output, check the application tracking page: http://ecs-hn1b-bd-cdp-edg-2:8188/applicationhistory/app/application_1591113812497_103797 Then click on links to logs of each attempt.
. Failing the application.
at com.tencent.angel.client.yarn.AngelYarnClient.updateMaster(AngelYarnClient.java:517)
at com.tencent.angel.client.yarn.AngelYarnClient.startPSServer(AngelYarnClient.java:170)
... 16 more
20/06/25 22:33:15 INFO spark.SparkContext: Invoking stop() from shutdown hook
20/06/25 22:33:15 INFO server.AbstractConnector: Stopped Spark@7a389761{HTTP/1.1,[http/1.1]}{0.0.0.0:4041}
20/06/25 22:33:15 INFO ui.SparkUI: Stopped Spark web UI at http://ecs-hn1a-xng-alg-rcmd-edg-1:4041
20/06/25 22:33:15 INFO cluster.YarnClientSchedulerBackend: Interrupting monitor thread
20/06/25 22:33:15 INFO cluster.YarnClientSchedulerBackend: Shutting down all executors
20/06/25 22:33:15 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Asking each executor to shut down
20/06/25 22:33:15 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices
(serviceOption=None,
services=List(),
started=false)
20/06/25 22:33:15 INFO cluster.YarnClientSchedulerBackend: Stopped
20/06/25 22:33:15 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/06/25 22:33:15 INFO memory.MemoryStore: MemoryStore cleared
20/06/25 22:33:15 INFO storage.BlockManager: BlockManager stopped
20/06/25 22:33:15 INFO storage.BlockManagerMaster: BlockManagerMaster stopped
20/06/25 22:33:15 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/06/25 22:33:15 INFO spark.SparkContext: Successfully stopped SparkContext
20/06/25 22:33:15 INFO util.ShutdownHookManager: Shutdown hook called
20/06/25 22:33:15 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-a689fb57-937d-4d2b-bf50-f59d08e9b6b8
20/06/25 22:33:15 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-2d255dca-b9e1-45ad-ba9d-6b8beae996e3
20/06/25 22:33:15 INFO client.AngelClient: stop the application
20/06/25 22:33:15 INFO client.AngelClient: master is null, just kill the application
20/06/25 22:33:15 INFO impl.YarnClientImpl: Killed application application_1591113812497_103797

@lzz156
Copy link

lzz156 commented Dec 11, 2020

请问问题解决了吗
我也遇到相同的问题了

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants