Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5028

Maps fail when io.sort.mb is set to high value

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.1.1, 2.0.3-alpha, 0.23.5
    • Fix Version/s: 1.2.0, 2.4.0
    • Component/s: None
    • Labels:
      None

      Description

      Verified the problem exists on branch-1 with the following configuration:

      Pseudo-dist mode: 2 maps/ 1 reduce, mapred.child.java.opts=-Xmx2048m, io.sort.mb=1280, dfs.block.size=2147483648

      Run teragen to generate 4 GB data
      Maps fail when you run wordcount on this configuration with the following error:

      java.io.IOException: Spill failed
      	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1031)
      	at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692)
      	at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
      	at org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:45)
      	at org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:34)
      	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
      	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
      	at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:396)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
      	at org.apache.hadoop.mapred.Child.main(Child.java:249)
      Caused by: java.io.EOFException
      	at java.io.DataInputStream.readInt(DataInputStream.java:375)
      	at org.apache.hadoop.io.IntWritable.readFields(IntWritable.java:38)
      	at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
      	at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
      	at org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:116)
      	at org.apache.hadoop.mapreduce.ReduceContext.nextKey(ReduceContext.java:92)
      	at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:175)
      	at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1505)
      	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1438)
      	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:855)
      	at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1346)
      
      1. mr-5028-branch1.patch
        3 kB
        Karthik Kambatla
      2. mr-5028-branch1.patch
        3 kB
        Karthik Kambatla
      3. mr-5028-branch1.patch
        2 kB
        Karthik Kambatla
      4. mr-5028-trunk.patch
        4 kB
        Karthik Kambatla
      5. mr-5028-trunk.patch
        4 kB
        Karthik Kambatla
      6. mr-5028-trunk.patch
        5 kB
        Karthik Kambatla
      7. repro-mr-5028.patch
        9 kB
        Karthik Kambatla
      8. MR-5028_testapp.patch
        11 kB
        Arun C Murthy
      9. mr-5028-1.patch
        16 kB
        Karthik Kambatla
      10. mr-5028-2.patch
        16 kB
        Karthik Kambatla
      11. mr-5028-3.patch
        20 kB
        Karthik Kambatla

        Issue Links

          Activity

          Gavin made changes -
          Reporter Karthik Kambatla [ kkambatl ] Karthik Kambatla [ kasha ]
          Gavin made changes -
          Assignee Karthik Kambatla [ kkambatl ] Karthik Kambatla [ kasha ]
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Vinod Kumar Vavilapalli made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 2.4.0 [ 12326141 ]
          Resolution Fixed [ 1 ]
          Karthik Kambatla (Inactive) made changes -
          Attachment mr-5028-3.patch [ 12633783 ]
          Karthik Kambatla (Inactive) made changes -
          Fix Version/s 1.2.0 [ 12321661 ]
          Vinod Kumar Vavilapalli made changes -
          Fix Version/s 1.2.0 [ 12321661 ]
          Karthik Kambatla (Inactive) made changes -
          Attachment mr-5028-2.patch [ 12633055 ]
          Karthik Kambatla (Inactive) made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Karthik Kambatla (Inactive) made changes -
          Attachment mr-5028-1.patch [ 12633012 ]
          Karthik Kambatla (Inactive) made changes -
          Fix Version/s 2.4.0 [ 12326141 ]
          Target Version/s 3.0.0, 0.23.11 [ 12320355, 12324663 ] 0.23.11, 2.4.0 [ 12324663, 12326141 ]
          Arun C Murthy made changes -
          Fix Version/s 2.4.0 [ 12326141 ]
          Fix Version/s 2.3.0 [ 12325257 ]
          Arun C Murthy made changes -
          Fix Version/s 2.3.0 [ 12325257 ]
          Fix Version/s 2.4.0 [ 12324590 ]
          Thomas Graves made changes -
          Target Version/s 1.2.0, 3.0.0, 2.1.0-beta, 0.23.10 [ 12321661, 12320355, 12324032, 12324662 ] 3.0.0, 2.4.0, 0.23.11 [ 12320355, 12324590, 12324663 ]
          Arun C Murthy made changes -
          Fix Version/s 2.3.0 [ 12324590 ]
          Fix Version/s 2.1.0-beta [ 12324032 ]
          Karthik Kambatla (Inactive) made changes -
          Link This issue relates to MAPREDUCE-5032 [ MAPREDUCE-5032 ]
          Thomas Graves made changes -
          Target Version/s 1.2.0, 3.0.0, 2.1.0-beta, 0.23.9 [ 12321661, 12320355, 12324032, 12324565 ] 1.2.0, 3.0.0, 2.1.0-beta, 0.23.10 [ 12321661, 12320355, 12324032, 12324662 ]
          Thomas Graves made changes -
          Target Version/s 1.2.0, 3.0.0, 2.0.5-beta, 0.23.8 [ 12321661, 12320355, 12324032, 12324140 ] 1.2.0, 3.0.0, 2.0.5-beta, 0.23.9 [ 12321661, 12320355, 12324032, 12324565 ]
          Arun C Murthy made changes -
          Attachment MR-5028_testapp.patch [ 12583158 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Matt Foley made changes -
          Target Version/s 3.0.0, 2.0.5-beta, 0.23.8 [ 12320355, 12324032, 12324140 ] 1.2.0, 3.0.0, 2.0.5-beta, 0.23.8 [ 12321661, 12320355, 12324032, 12324140 ]
          Karthik Kambatla (Inactive) made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Karthik Kambatla (Inactive) made changes -
          Attachment org.apache.hadoop.mapreduce.v2.TestMRJobs-output.txt [ 12574367 ]
          Karthik Kambatla (Inactive) made changes -
          Attachment repro-mr-5028.patch [ 12577110 ]
          Thomas Graves made changes -
          Target Version/s 1.2.0, 0.23.7, 2.0.5-beta [ 12321661, 12323954, 12324032 ] 3.0.0, 2.0.5-beta, 0.23.8 [ 12320355, 12324032, 12324140 ]
          Thomas Graves made changes -
          Fix Version/s 0.23.7 [ 12323954 ]
          Arun C Murthy made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Karthik Kambatla (Inactive) made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Karthik Kambatla (Inactive) made changes -
          Attachment mr-5028-trunk.patch [ 12574490 ]
          Karthik Kambatla (Inactive) made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Karthik Kambatla (Inactive) made changes -
          Status Reopened [ 4 ] Patch Available [ 10002 ]
          Karthik Kambatla (Inactive) made changes -
          Attachment mr-5028-trunk.patch [ 12574399 ]
          Robert Joseph Evans made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Karthik Kambatla (Inactive) made changes -
          Assignee Alejandro Abdelnur [ tucu00 ] Karthik Kambatla [ kkambatl ]
          Karthik Kambatla (Inactive) made changes -
          Attachment org.apache.hadoop.mapreduce.v2.TestMRJobs-output.txt [ 12574367 ]
          Thomas Graves made changes -
          Fix Version/s 0.23.7 [ 12323954 ]
          Alejandro Abdelnur made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 2.0.5-beta [ 12324032 ]
          Resolution Fixed [ 1 ]
          Alejandro Abdelnur made changes -
          Assignee Karthik Kambatla [ kkambatl ] Alejandro Abdelnur [ tucu00 ]
          Karthik Kambatla (Inactive) made changes -
          Fix Version/s 1.2.0 [ 12321661 ]
          Karthik Kambatla (Inactive) made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Karthik Kambatla (Inactive) made changes -
          Attachment mr-5028-trunk.patch [ 12571997 ]
          Karthik Kambatla (Inactive) made changes -
          Affects Version/s 0.23.5 [ 12323312 ]
          Affects Version/s 2.0.3-alpha [ 12323275 ]
          Target Version/s 1.2.0 [ 12321661 ] 1.2.0, 0.23.7, 2.0.4-beta [ 12321661, 12323954, 12324032 ]
          Karthik Kambatla (Inactive) made changes -
          Attachment mr-5028-branch1.patch [ 12571715 ]
          Karthik Kambatla (Inactive) made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Karthik Kambatla (Inactive) made changes -
          Attachment mr-5028-branch1.patch [ 12571090 ]
          Karthik Kambatla (Inactive) made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Karthik Kambatla (Inactive) made changes -
          Description Verified the problem exists on branch-1 with the following configuration:

          Pseudo-dist mode: 2 maps/ 1 reduce, mapred.child.java.opts=-Xmx2048m, io.sort.mb=1280, dfs.block.size=2147483648

          Run teragen to generate 4 GB data
          Maps fail when you run wordcount on this configuration with the following error:
          {noformat}
          java.io.IOException: Spill failed
          at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1031)
          at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692)
          at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
          at org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:45)
          at org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:34)
          at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
          at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
          at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
          at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:396)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
          at org.apache.hadoop.mapred.Child.main(Child.java:249)
          Caused by: java.io.EOFException
          at java.io.DataInputStream.readInt(DataInputStream.java:375)
          at org.apache.hadoop.io.IntWritable.readFields(IntWritable.java:38)
          at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
          at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
          at org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:116)
          at org.apache.hadoop.mapreduce.ReduceContext.nextKey(ReduceContext.java:92)
          at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:175)
          at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1505)
          at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1438)
          at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:855)
          at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1346)
          {noformat}

          Marked branch-0.23 and branch-2 also because the offending code seems to exist there too.
          Verified the problem exists on branch-1 with the following configuration:

          Pseudo-dist mode: 2 maps/ 1 reduce, mapred.child.java.opts=-Xmx2048m, io.sort.mb=1280, dfs.block.size=2147483648

          Run teragen to generate 4 GB data
          Maps fail when you run wordcount on this configuration with the following error:
          {noformat}
          java.io.IOException: Spill failed
          at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1031)
          at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:692)
          at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
          at org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:45)
          at org.apache.hadoop.examples.WordCount$TokenizerMapper.map(WordCount.java:34)
          at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
          at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:766)
          at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
          at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:396)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
          at org.apache.hadoop.mapred.Child.main(Child.java:249)
          Caused by: java.io.EOFException
          at java.io.DataInputStream.readInt(DataInputStream.java:375)
          at org.apache.hadoop.io.IntWritable.readFields(IntWritable.java:38)
          at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:67)
          at org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:40)
          at org.apache.hadoop.mapreduce.ReduceContext.nextKeyValue(ReduceContext.java:116)
          at org.apache.hadoop.mapreduce.ReduceContext.nextKey(ReduceContext.java:92)
          at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:175)
          at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1505)
          at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1438)
          at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:855)
          at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1346)
          {noformat}
          Karthik Kambatla (Inactive) made changes -
          Link This issue is related to MAPREDUCE-5031 [ MAPREDUCE-5031 ]
          Karthik Kambatla (Inactive) made changes -
          Target Version/s 1.2.0, 0.23.7, 2.0.4-beta [ 12321661, 12323954, 12324032 ] 1.2.0 [ 12321661 ]
          Karthik Kambatla (Inactive) made changes -
          Affects Version/s 2.0.3-alpha [ 12323275 ]
          Affects Version/s 0.23.5 [ 12323312 ]
          Karthik Kambatla (Inactive) made changes -
          Field Original Value New Value
          Attachment mr-5028-branch1.patch [ 12570924 ]
          Karthik Kambatla (Inactive) created issue -

            People

            • Assignee:
              Karthik Kambatla
              Reporter:
              Karthik Kambatla
            • Votes:
              0 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development