Details
-
Bug
-
Status: Closed
-
Blocker
-
Resolution: Cannot Reproduce
-
0.23.0
-
None
Description
Intermittently DFSIO tests are failing either in "read" or "write" operations.
$HADOOP_COMMON_HOME/bin/hadoop -config $HADOOP_CONF_DIR jar $HADOOP_MAPRED_HOME/hadoop-mapreduce-test*.jar TestDFSIO -read -nrFiles 36 -fileSize 320
11/09/27 03:21:03 INFO fs.TestDFSIO: TestDFSIO.0.0.6
11/09/27 03:21:03 INFO fs.TestDFSIO: nrFiles = 36
11/09/27 03:21:03 INFO fs.TestDFSIO: fileSize (MB) = 320.0
11/09/27 03:21:03 INFO fs.TestDFSIO: bufferSize = 1000000
11/09/27 03:21:03 INFO fs.TestDFSIO: baseDir = /benchmarks/TestDFSIO
11/09/27 03:21:03 INFO fs.TestDFSIO: creating control file: 335544320 bytes, 36 files
11/09/27 03:21:05 INFO fs.TestDFSIO: created control files for: 36 files
..
..
..
..
..
11/09/27 03:24:04 INFO mapreduce.Job: map 88% reduce 25%
11/09/27 03:24:05 INFO mapreduce.Job: map 90% reduce 25%
11/09/27 03:24:06 INFO mapreduce.Job: map 92% reduce 25%
11/09/27 03:24:09 INFO mapreduce.Job: map 93% reduce 25%
11/09/27 03:24:10 INFO mapreduce.Job: map 93% reduce 26%
11/09/27 03:24:42 INFO mapreduce.Job: map 94% reduce 26%
11/09/27 03:24:43 INFO mapreduce.Job: map 94% reduce 27%
11/09/27 03:24:56 INFO mapreduce.Job: map 95% reduce 27%
11/09/27 03:24:58 INFO mapreduce.Job: map 95% reduce 28%
11/09/27 03:25:13 INFO mapreduce.Job: map 96% reduce 28%
11/09/27 03:25:15 INFO mapreduce.Job: map 97% reduce 28%
11/09/27 03:25:16 INFO mapreduce.Job: map 97% reduce 30%
11/09/27 03:25:16 INFO mapreduce.Job: map 98% reduce 30%
11/09/27 03:25:16 INFO mapreduce.Job: Job job_1317092846056_0012 failed with state FAILED
11/09/27 03:25:16 INFO mapreduce.Job: Counters: 44
File System Counters
FILE: BYTES_READ=20931
FILE: BYTES_WRITTEN=2192966
FILE: READ_OPS=0
FILE: LARGE_READ_OPS=0
FILE: WRITE_OPS=0
HDFS: BYTES_READ=10454250228
HDFS: BYTES_WRITTEN=0
HDFS: READ_OPS=172
HDFS: LARGE_READ_OPS=0
HDFS: WRITE_OPS=0
org.apache.hadoop.mapreduce.JobCounter
NUM_FAILED_MAPS=6
TOTAL_LAUNCHED_MAPS=44
TOTAL_LAUNCHED_REDUCES=1
DATA_LOCAL_MAPS=10
RACK_LOCAL_MAPS=34
SLOTS_MILLIS_MAPS=2340307
org.apache.hadoop.mapreduce.TaskCounter
MAP_INPUT_RECORDS=36
MAP_OUTPUT_RECORDS=140
MAP_OUTPUT_BYTES=2103
MAP_OUTPUT_MATERIALIZED_BYTES=2579
SPLIT_RAW_BYTES=5210
COMBINE_INPUT_RECORDS=0
COMBINE_OUTPUT_RECORDS=0
REDUCE_INPUT_GROUPS=0
REDUCE_SHUFFLE_BYTES=3054
REDUCE_INPUT_RECORDS=0
REDUCE_OUTPUT_RECORDS=0
SPILLED_RECORDS=140
SHUFFLED_MAPS=33
FAILED_SHUFFLE=0
MERGED_MAP_OUTPUTS=0
GC_TIME_MILLIS=59933
CPU_MILLISECONDS=159470
PHYSICAL_MEMORY_BYTES=11310596096
VIRTUAL_MEMORY_BYTES=31425290240
COMMITTED_HEAP_BYTES=12728664064
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
BYTES_READ=4058
org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter
BYTES_WRITTEN=0
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:781)
at org.apache.hadoop.fs.TestDFSIO.runIOTest(TestDFSIO.java:340)
at org.apache.hadoop.fs.TestDFSIO.readTest(TestDFSIO.java:418)
at org.apache.hadoop.fs.TestDFSIO.run(TestDFSIO.java:522)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83)
at org.apache.hadoop.fs.TestDFSIO.main(TestDFSIO.java:445)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:189)