Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-3334 Tez Custom Shuffle Handler
  3. TEZ-3633

Implement keep-alive timeout in tez shuffle handler

Attach filesAttach ScreenshotVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.9.0
    • None
    • None

    Description

      MAPREDUCE-5787 which added keep-alive to mapreduce shuffle handler was not fully functional as despite advertising keep-alive option and adding the header to the response, all connections were closed immediately after write. This reduced the performance of certain fetches as now time is spent requesting a second get to the same serve, only for that server to reset the connection forcing the client to reestablish the connection on another port. The details of this is hidden behind HttpURLConnection and doesn't show in any log file at default logging level. However TCP sniffing does show errant behavior.

      Attachments

        1. TEZ-3633.1.patch
          5 kB
          Jonathan Turner Eagles
        2. TEZ-3633.2.patch
          14 kB
          Jonathan Turner Eagles
        3. with_hadoop_2.7.3.png
          388 kB
          Rajesh Balamohan

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            jeagles Jonathan Turner Eagles
            jeagles Jonathan Turner Eagles
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment