Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5279

Jobs can deadlock if headroom is limited by cpu instead of memory

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 2.0.3-alpha
    • 2.6.0
    • mrv2, scheduler
    • None
    • Reviewed

    Description

      YARN-2 imported cpu dimension scheduling, but MR RMContainerAllocator doesn't take into account virtual cores while scheduling reduce tasks.
      This may cause more reduce tasks to be scheduled because memory is enough. And on a small cluster, this will end with deadlock, all running containers are reduce tasks but map phase is not finished.

      Attachments

        1. MAPREDUCE-5279-v2.patch
          27 kB
          Peng Zhang
        2. MAPREDUCE-5279-branch-2.patch
          35 kB
          Zhijie Shen
        3. MAPREDUCE-5279.patch
          28 kB
          Peng Zhang
        4. apache-mapreduce-5279.7.patch
          35 kB
          Varun Vasudev
        5. apache-mapreduce-5279.6.patch
          35 kB
          Varun Vasudev
        6. apache-mapreduce-5279.5.patch
          38 kB
          Varun Vasudev
        7. apache-mapreduce-5279.4.patch
          38 kB
          Varun Vasudev
        8. apache-mapreduce-5279.3.patch
          38 kB
          Varun Vasudev

        Activity

          People

            peng.zhang Peng Zhang
            peng.zhang Peng Zhang
            Votes:
            0 Vote for this issue
            Watchers:
            18 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: