Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7244

ShuffleHandler is not aware of disks that are added

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 2.8.3, 3.0.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The ShuffleHandler permanently remembers the list of "good" disks on NM startup. If disks later are added to the node then map tasks will start using them but the ShuffleHandler will not be aware of them. The end result is that the data cannot be shuffled from the node leading to fetch failures and re-runs of the map tasks.

        Attachments

        1. YARN-7244.001.patch
          13 kB
          Kuhu Shukla
        2. YARN-7244.002.patch
          14 kB
          Kuhu Shukla
        3. YARN-7244.003.patch
          18 kB
          Kuhu Shukla
        4. YARN-7244.004.patch
          18 kB
          Kuhu Shukla
        5. YARN-7244.005.patch
          27 kB
          Kuhu Shukla
        6. YARN-7244.006.patch
          30 kB
          Kuhu Shukla
        7. YARN-7244.007.patch
          33 kB
          Kuhu Shukla
        8. YARN-7244.008.patch
          33 kB
          Kuhu Shukla
        9. YARN-7244.009.patch
          33 kB
          Kuhu Shukla
        10. YARN-7244.010.patch
          32 kB
          Kuhu Shukla
        11. YARN-7244.011.patch
          32 kB
          Kuhu Shukla
        12. YARN-7244.012.patch
          32 kB
          Kuhu Shukla
        13. YARN-7244.013.patch
          32 kB
          Kuhu Shukla
        14. YARN-7244-branch-2.8.001.patch
          38 kB
          Kuhu Shukla
        15. YARN-7244-branch-2.8.002.patch
          38 kB
          Kuhu Shukla

          Activity

            People

            • Assignee:
              kshukla Kuhu Shukla
              Reporter:
              kshukla Kuhu Shukla
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: