hadoop map failure

[soaphee]

I'm using Crossbow with Hadoop 2.4 on a cluster of 3, and get this error:

Crossbow job

Hadoop streaming commands in: /tmp/crossbow-9ZzYhbmcKc/invoke.scripts/cb.7742.hadoop.sh

Running...

Stage 1 of 4. Preprocess

Fri Jun 27 13:31:44 CDT 2014
14/06/27 13:31:45 WARN streaming.StreamJob: -file option is deprecated, please use generic option -files instead.
OpenJDK 64-Bit Server VM warning: You have loaded library /usr/local/hadoop/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now.
It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'.
14/06/27 13:31:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
packageJobJar: [/home/hadoop/crossbow-1.2.0/Get.pm, /home/hadoop/crossbow-1.2.0/Counters.pm, /home/hadoop/crossbow-1.2.0/Util.pm, /home/hadoop/crossbow-1.2.0/Tools.pm, /home/hadoop/crossbow-1.2.0/AWS.pm, /tmp/hadoop-hadoop/hadoop-unjar8648399505312935021/] [] /tmp/streamjob8287394335394764183.jar tmpDir=null
14/06/27 13:31:47 INFO client.RMProxy: Connecting to ResourceManager at hadoopmaster/10.53.14.117:8032
14/06/27 13:31:47 INFO client.RMProxy: Connecting to ResourceManager at hadoopmaster/10.53.14.117:8032
14/06/27 13:31:48 INFO mapred.FileInputFormat: Total input paths to process : 1
14/06/27 13:31:48 INFO mapreduce.JobSubmitter: number of splits:11
14/06/27 13:31:48 INFO Configuration.deprecation: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/06/27 13:31:48 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/06/27 13:31:48 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1403714184227_0009
14/06/27 13:31:49 INFO impl.YarnClientImpl: Submitted application application_1403714184227_0009
14/06/27 13:31:49 INFO mapreduce.Job: The url to track the job: http://50.23.84.174-static.reverse.softlayer.com:8088/proxy/application_1403714184227_0009/
14/06/27 13:31:49 INFO mapreduce.Job: Running job: job_1403714184227_0009
14/06/27 13:31:55 INFO mapreduce.Job: Job job_1403714184227_0009 running in uber mode : false
14/06/27 13:31:55 INFO mapreduce.Job: map 0% reduce 0%
14/06/27 13:32:08 INFO mapreduce.Job: map 9% reduce 0%
14/06/27 13:32:09 INFO mapreduce.Job: map 18% reduce 0%
14/06/27 13:32:11 INFO mapreduce.Job: map 27% reduce 0%
14/06/27 13:32:12 INFO mapreduce.Job: map 36% reduce 0%
14/06/27 13:32:13 INFO mapreduce.Job: map 55% reduce 0%
14/06/27 13:32:15 INFO mapreduce.Job: map 73% reduce 0%
14/06/27 13:32:16 INFO mapreduce.Job: map 82% reduce 0%
14/06/27 13:32:17 INFO mapreduce.Job: map 100% reduce 0%
14/06/27 13:34:23 INFO mapreduce.Job: Task Id : attempt_1403714184227_0009_m_000001_0, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

14/06/27 13:34:24 INFO mapreduce.Job: map 91% reduce 0%
14/06/27 13:34:55 INFO mapreduce.Job: map 100% reduce 0%
14/06/27 13:34:58 INFO mapreduce.Job: Task Id : attempt_1403714184227_0009_m_000004_0, Status : FAILED
Error: java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2
at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320)
at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533)
at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

...

14/06/27 13:41:24 INFO mapreduce.Job: map 91% reduce 0%
14/06/27 13:41:31 INFO mapreduce.Job: map 100% reduce 0%
14/06/27 13:41:35 INFO mapreduce.Job: Job job_1403714184227_0009 failed with state FAILED due to: Task failed task_1403714184227_0009_m_000009
Job failed as tasks failed. failedMaps:1 failedReduces:0

14/06/27 13:41:38 INFO mapreduce.Job: Counters: 34
File System Counters
FILE: Number of bytes read=0
FILE: Number of bytes written=96812
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=1629
HDFS: Number of bytes written=6
HDFS: Number of read operations=5
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Failed map tasks=24
Killed map tasks=14
Launched map tasks=39
Other local map tasks=39
Total time spent by all maps in occupied slots (ms)=7474134
Total time spent by all reduces in occupied slots (ms)=0
Total time spent by all map tasks (ms)=7474134
Total vcore-seconds taken by all map tasks=7474134
Total megabyte-seconds taken by all map tasks=7653513216
Map-Reduce Framework
Map input records=1
Map output records=1
Input split bytes=127
Spilled Records=0
Failed Shuffles=0
Merged Map outputs=0
GC time elapsed (ms)=26
CPU time spent (ms)=800
Physical memory (bytes) snapshot=136933376
Virtual memory (bytes) snapshot=1168240640
Total committed heap usage (bytes)=85458944
Short read preprocessor
Comment lines=1
Warnings=0
File Input Format Counters
Bytes Read=1502
File Output Format Counters
Bytes Written=6
14/06/27 13:41:38 ERROR streaming.StreamJob: Job not Successful!
Streaming Command Failed!
Non-zero exitlevel from Preprocess stage

[ShradhaBharne]

Is this error solved? I am facing the same issue currently