Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5279

Jobs can deadlock if headroom is limited by cpu instead of memory

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.3-alpha
    • Fix Version/s: 2.6.0
    • Component/s: mrv2, scheduler
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      YARN-2 imported cpu dimension scheduling, but MR RMContainerAllocator doesn't take into account virtual cores while scheduling reduce tasks.
      This may cause more reduce tasks to be scheduled because memory is enough. And on a small cluster, this will end with deadlock, all running containers are reduce tasks but map phase is not finished.

        Attachments

        1. MAPREDUCE-5279-branch-2.patch
          35 kB
          Zhijie Shen
        2. apache-mapreduce-5279.7.patch
          35 kB
          Varun Vasudev
        3. apache-mapreduce-5279.6.patch
          35 kB
          Varun Vasudev
        4. apache-mapreduce-5279.5.patch
          38 kB
          Varun Vasudev
        5. apache-mapreduce-5279.4.patch
          38 kB
          Varun Vasudev
        6. apache-mapreduce-5279.3.patch
          38 kB
          Varun Vasudev
        7. MAPREDUCE-5279-v2.patch
          27 kB
          Peng Zhang
        8. MAPREDUCE-5279.patch
          28 kB
          Peng Zhang

          Activity

            People

            • Assignee:
              peng.zhang Peng Zhang
              Reporter:
              peng.zhang Peng Zhang
            • Votes:
              0 Vote for this issue
              Watchers:
              19 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: