Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-7244

ShuffleHandler is not aware of disks that are added

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.9.0, 2.8.3, 3.0.0
    • None
    • None
    • Reviewed

    Description

      The ShuffleHandler permanently remembers the list of "good" disks on NM startup. If disks later are added to the node then map tasks will start using them but the ShuffleHandler will not be aware of them. The end result is that the data cannot be shuffled from the node leading to fetch failures and re-runs of the map tasks.

      Attachments

        1. YARN-7244.001.patch
          13 kB
          Kuhu Shukla
        2. YARN-7244.002.patch
          14 kB
          Kuhu Shukla
        3. YARN-7244.003.patch
          18 kB
          Kuhu Shukla
        4. YARN-7244.004.patch
          18 kB
          Kuhu Shukla
        5. YARN-7244.005.patch
          27 kB
          Kuhu Shukla
        6. YARN-7244.006.patch
          30 kB
          Kuhu Shukla
        7. YARN-7244.007.patch
          33 kB
          Kuhu Shukla
        8. YARN-7244.008.patch
          33 kB
          Kuhu Shukla
        9. YARN-7244.009.patch
          33 kB
          Kuhu Shukla
        10. YARN-7244.010.patch
          32 kB
          Kuhu Shukla
        11. YARN-7244.011.patch
          32 kB
          Kuhu Shukla
        12. YARN-7244.012.patch
          32 kB
          Kuhu Shukla
        13. YARN-7244.013.patch
          32 kB
          Kuhu Shukla
        14. YARN-7244-branch-2.8.001.patch
          38 kB
          Kuhu Shukla
        15. YARN-7244-branch-2.8.002.patch
          38 kB
          Kuhu Shukla

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            kshukla Kuhu Shukla
            kshukla Kuhu Shukla
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment