Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6631

shuffle handler would benefit from per-local-dir threads

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 2.7.2, 3.0.0-alpha1
    • None
    • nodemanager
    • None

    Description

      Jason Darrell Lowe and I discussed this while investigating I/O starvation we have been seeing on our clusters lately (possibly amplified by increased tez workloads).

      If a particular disk is being slow, it is very likely that all shuffle netty threads will be blocked on the read side of sendfile(). (sendfile() is asynchronous on the outbound socket side, but not on the read side.) This causes the entire shuffle subsystem to slow down.

      It seems like we could make the netty threads more asynchronous by introducing a small set of threads per local-dir that are responsible for the actual sendfile() invocations.

      This would not only improve shuffles that span drives, but also improve situations where there is a single large shuffle from a single local-dir. It would allow other drives to continue serving shuffle requests, AND avoid a large number of readers (2X number_of_cores by default) all fighting for the same drive, which becomes unfair to everything else on the system.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            nroberts Nathan Roberts

            Dates

              Created:
              Updated:

              Slack

                Issue deployment