Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.203.0, 1.0.3
    • Fix Version/s: 1.0.4
    • Component/s: performance, tasktracker
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      There is a significant (up to 3x) performance regression in shuffle (vs 0.20.2) in the Hadoop 1.x series. Most noticeable with high-end switches.

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        15d 7h 52m 1 Suresh Srinivas 20/Jul/12 19:13
        Resolved Resolved Closed Closed
        89d 11m 1 Matt Foley 17/Oct/12 19:25
        Matt Foley made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Matt Foley added a comment -

        Closed upon release of Hadoop-1.0.4.

        Show
        Matt Foley added a comment - Closed upon release of Hadoop-1.0.4.
        Matt Foley made changes -
        Fix Version/s 1.0.4 [ 12323325 ]
        Fix Version/s 1.1.0 [ 12317960 ]
        Matt Foley made changes -
        Target Version/s 1.1.0 [ 12317960 ] 1.0.4 [ 12323325 ]
        Matt Foley made changes -
        Fix Version/s 1.1.0 [ 12317960 ]
        Hide
        Matt Foley added a comment -

        Committed to branch-1.1 and branch-1.0. Thanks, Luke!

        Show
        Matt Foley added a comment - Committed to branch-1.1 and branch-1.0. Thanks, Luke!
        Hide
        Luke Lu added a comment -

        @Suresh, 0.20.2 use jetty 6.1.14, which doesn't have this particular issue. This is a workaround specifically for jetty 6.1.26 and harmless for other versions.

        Show
        Luke Lu added a comment - @Suresh, 0.20.2 use jetty 6.1.14, which doesn't have this particular issue. This is a workaround specifically for jetty 6.1.26 and harmless for other versions.
        Hide
        Suresh Srinivas added a comment -

        Nicholas, can you please verify if this is an issue for WebHDFS (both on 1.0 and trunk?)

        Show
        Suresh Srinivas added a comment - Nicholas, can you please verify if this is an issue for WebHDFS (both on 1.0 and trunk?)
        Suresh Srinivas made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Resolution Fixed [ 1 ]
        Hide
        Suresh Srinivas added a comment -

        I committed the patch to branch-1 and for 1.1.0.

        Thank you Luke for the patch. Thank you John Poelman and Shreyas Subramanya for noticing the issue and verifying the fix.

        Show
        Suresh Srinivas added a comment - I committed the patch to branch-1 and for 1.1.0. Thank you Luke for the patch. Thank you John Poelman and Shreyas Subramanya for noticing the issue and verifying the fix.
        Hide
        Suresh Srinivas added a comment -

        Given the code is the same, this may not be regression compared to 0.20.2. However, this is straightforward change, and given the link you posted, I am +1 on committing this.

        Show
        Suresh Srinivas added a comment - Given the code is the same, this may not be regression compared to 0.20.2. However, this is straightforward change, and given the link you posted, I am +1 on committing this.
        Suresh Srinivas made changes -
        Component/s performance [ 12316500 ]
        Hide
        Luke Lu added a comment -

        The crux of the problem is jetty 6.1.26's internal buffer management with transfer-encoding = chunked. The issue was also independently discovered and analyzed elsewhere: http://goo.gl/Wln7Z

        Show
        Luke Lu added a comment - The crux of the problem is jetty 6.1.26's internal buffer management with transfer-encoding = chunked. The issue was also independently discovered and analyzed elsewhere: http://goo.gl/Wln7Z
        Arun C Murthy made changes -
        Comment [ Oops: the link should be: http://goo.gl/Wln7Z ]
        Arun C Murthy made changes -
        Comment [ The crux of the problem is jetty 6.1.26's internal buffer management with transfer-encoding = trunked. The issue was also independently discovered and analyzed elsewhere: http://goo.gl/dtM4D

        The patch is trivial. ]
        Luke Lu made changes -
        Field Original Value New Value
        Attachment mapreduce-4399-branch-1.patch [ 12535444 ]
        Hide
        Luke Lu added a comment -

        Thanks to John Poelman and Shreyas Subramanya of IBM BigInsights performance QA for noticing the issue and verifying my fix.

        Show
        Luke Lu added a comment - Thanks to John Poelman and Shreyas Subramanya of IBM BigInsights performance QA for noticing the issue and verifying my fix.
        Luke Lu created issue -

          People

          • Assignee:
            Luke Lu
            Reporter:
            Luke Lu
          • Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development