Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5744

Job hangs because RMContainerAllocator$AssignedRequests.preemptReduce() violates the comparator contract

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.0.5-alpha
    • Fix Version/s: 0.23.11, 2.3.0
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      We ran into a situation where tasks are not getting assigned because RMContainerAllocator$AssignedRequests.preemptReduce() fails repeatedly with the following exception:

      2014-02-06 16:43:45,183 ERROR [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: ERROR IN CONTACTING RM.
      java.lang.IllegalArgumentException: Comparison method violates its general contract!
           at java.util.TimSort.mergeLo(TimSort.java:747)
           at java.util.TimSort.mergeAt(TimSort.java:483)
           at java.util.TimSort.mergeCollapse(TimSort.java:408)
           at java.util.TimSort.sort(TimSort.java:214)
           at java.util.TimSort.sort(TimSort.java:173)
           at java.util.Arrays.sort(Arrays.java:659)
           at java.util.Collections.sort(Collections.java:217)
           at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator$AssignedRequests.preemptReduce(RMContainerAllocator.java:1106)
           at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.preemptReducesIfNeeded(RMContainerAllocator.java:416)
           at org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.heartbeat(RMContainerAllocator.java:230)
           at org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator$1.run(RMCommunicator.java:252)
           at java.lang.Thread.run(Thread.java:744)
      

      It is because the comparator that's defined in this method does not abide by the contract, specifically if p == 0.

      Comparator.compare(): http://docs.oracle.com/javase/7/docs/api/java/util/Comparator.html#compare(T, T)

        Attachments

        1. MAPREDUCE-5744.v02.patch
          1 kB
          Gera Shegalov

          Activity

            People

            • Assignee:
              jira.shegalov Gera Shegalov
              Reporter:
              sjlee0 Sangjin Lee
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: