Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6850

Shuffle Handler keep-alive connections are closed from the server side

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When performance testing tez shuffle handler (TEZ-3334), it was noticed the keep-alive connections are closed from the server-side. The client silently recovers and logs the connection as keep-alive, despite reestablishing a connection. This jira aims to remove the close from the server side, fixing the bug preventing keep-alive connections.

      1. With_Patch.png
        501 kB
        Rajesh Balamohan
      2. With_Patch_withData.png
        482 kB
        Rajesh Balamohan
      3. With_Issue.png
        224 kB
        Rajesh Balamohan
      4. MAPREDUCE-6850.7.patch
        13 kB
        Jonathan Eagles
      5. MAPREDUCE-6850.6.patch
        12 kB
        Jonathan Eagles
      6. MAPREDUCE-6850.5.patch
        11 kB
        Jonathan Eagles
      7. MAPREDUCE-6850.4.patch
        10 kB
        Jonathan Eagles
      8. MAPREDUCE-6850.3.patch
        10 kB
        Jonathan Eagles
      9. MAPREDUCE-6850.2.patch
        10 kB
        Jonathan Eagles
      10. MAPREDUCE-6850.1.patch
        7 kB
        Jonathan Eagles

        Issue Links

          Activity

          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11497 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11497/)
          MAPREDUCE-6850. Shuffle Handler keep-alive connections are closed from (jlowe: rev c8bd5fc7a86f9890ceaa37a89491ab650e7e9a64)

          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java
          • (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11497 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11497/ ) MAPREDUCE-6850 . Shuffle Handler keep-alive connections are closed from (jlowe: rev c8bd5fc7a86f9890ceaa37a89491ab650e7e9a64) (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/test/java/org/apache/hadoop/mapred/TestShuffleHandler.java (edit) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle/src/main/java/org/apache/hadoop/mapred/ShuffleHandler.java
          Hide
          jlowe Jason Lowe added a comment -

          Thanks to Jonathan for the contribution and to Rajesh for additional review! I committed this to trunk, branch-2, and branch-2.8.

          Show
          jlowe Jason Lowe added a comment - Thanks to Jonathan for the contribution and to Rajesh for additional review! I committed this to trunk, branch-2, and branch-2.8.
          Hide
          jlowe Jason Lowe added a comment -

          +1 lgtm. Committing this.

          Show
          jlowe Jason Lowe added a comment - +1 lgtm. Committing this.
          Hide
          jeagles Jonathan Eagles added a comment -

          Jason Lowe, posted patch that moves timeout to a private state final variable.

          Show
          jeagles Jonathan Eagles added a comment - Jason Lowe , posted patch that moves timeout to a private state final variable.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 24s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 17m 39s trunk passed
          +1 compile 0m 21s trunk passed
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 0m 22s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          +1 findbugs 0m 32s trunk passed
          +1 javadoc 0m 18s trunk passed
          +1 mvninstall 0m 21s the patch passed
          +1 compile 0m 19s the patch passed
          +1 javac 0m 19s the patch passed
          +1 checkstyle 0m 12s the patch passed
          +1 mvnsite 0m 20s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 44s the patch passed
          +1 javadoc 0m 14s the patch passed
          +1 unit 0m 24s hadoop-mapreduce-client-shuffle in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          24m 8s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861096/MAPREDUCE-6850.7.patch
          JIRA Issue MAPREDUCE-6850
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux cc2af3a8a738 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 640ba1d
          Default Java 1.8.0_121
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6937/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6937/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 17m 39s trunk passed +1 compile 0m 21s trunk passed +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 22s trunk passed +1 mvneclipse 0m 17s trunk passed +1 findbugs 0m 32s trunk passed +1 javadoc 0m 18s trunk passed +1 mvninstall 0m 21s the patch passed +1 compile 0m 19s the patch passed +1 javac 0m 19s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 20s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 44s the patch passed +1 javadoc 0m 14s the patch passed +1 unit 0m 24s hadoop-mapreduce-client-shuffle in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 24m 8s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861096/MAPREDUCE-6850.7.patch JIRA Issue MAPREDUCE-6850 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux cc2af3a8a738 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 640ba1d Default Java 1.8.0_121 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6937/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6937/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for updating the patch! Patch looks good, only a minor nit that "timeout" is now used in multiple places and would be good to have as a private static final String.

          Show
          jlowe Jason Lowe added a comment - Thanks for updating the patch! Patch looks good, only a minor nit that "timeout" is now used in multiple places and would be good to have as a private static final String.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 14m 34s trunk passed
          +1 compile 0m 16s trunk passed
          +1 checkstyle 0m 12s trunk passed
          +1 mvnsite 0m 18s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 0m 26s trunk passed
          +1 javadoc 0m 14s trunk passed
          +1 mvninstall 0m 16s the patch passed
          +1 compile 0m 14s the patch passed
          +1 javac 0m 14s the patch passed
          +1 checkstyle 0m 9s the patch passed
          +1 mvnsite 0m 16s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 33s the patch passed
          +1 javadoc 0m 10s the patch passed
          +1 unit 0m 20s hadoop-mapreduce-client-shuffle in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          19m 47s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861088/MAPREDUCE-6850.6.patch
          JIRA Issue MAPREDUCE-6850
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 3707bfe53a74 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 640ba1d
          Default Java 1.8.0_121
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6936/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6936/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 14m 34s trunk passed +1 compile 0m 16s trunk passed +1 checkstyle 0m 12s trunk passed +1 mvnsite 0m 18s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 0m 26s trunk passed +1 javadoc 0m 14s trunk passed +1 mvninstall 0m 16s the patch passed +1 compile 0m 14s the patch passed +1 javac 0m 14s the patch passed +1 checkstyle 0m 9s the patch passed +1 mvnsite 0m 16s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 33s the patch passed +1 javadoc 0m 10s the patch passed +1 unit 0m 20s hadoop-mapreduce-client-shuffle in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 19m 47s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861088/MAPREDUCE-6850.6.patch JIRA Issue MAPREDUCE-6850 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3707bfe53a74 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 640ba1d Default Java 1.8.0_121 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6936/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6936/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 34s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 42s trunk passed
          +1 compile 0m 16s trunk passed
          +1 checkstyle 0m 12s trunk passed
          +1 mvnsite 0m 17s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 23s trunk passed
          +1 javadoc 0m 13s trunk passed
          +1 mvninstall 0m 13s the patch passed
          +1 compile 0m 13s the patch passed
          +1 javac 0m 13s the patch passed
          +1 checkstyle 0m 9s the patch passed
          +1 mvnsite 0m 13s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 26s the patch passed
          +1 javadoc 0m 10s the patch passed
          -1 unit 0m 17s hadoop-mapreduce-client-shuffle in the patch failed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          17m 40s



          Reason Tests
          Failed junit tests hadoop.mapred.TestShuffleHandler



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861085/MAPREDUCE-6850.5.patch
          JIRA Issue MAPREDUCE-6850
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux eb30a5823f05 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 13c766b
          Default Java 1.8.0_121
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6935/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6935/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6935/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6935/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 34s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 42s trunk passed +1 compile 0m 16s trunk passed +1 checkstyle 0m 12s trunk passed +1 mvnsite 0m 17s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 23s trunk passed +1 javadoc 0m 13s trunk passed +1 mvninstall 0m 13s the patch passed +1 compile 0m 13s the patch passed +1 javac 0m 13s the patch passed +1 checkstyle 0m 9s the patch passed +1 mvnsite 0m 13s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 26s the patch passed +1 javadoc 0m 10s the patch passed -1 unit 0m 17s hadoop-mapreduce-client-shuffle in the patch failed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 17m 40s Reason Tests Failed junit tests hadoop.mapred.TestShuffleHandler Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861085/MAPREDUCE-6850.5.patch JIRA Issue MAPREDUCE-6850 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux eb30a5823f05 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 13c766b Default Java 1.8.0_121 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6935/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6935/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6935/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6935/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jeagles Jonathan Eagles added a comment -

          Jason Lowe, added state to the timeout handler (one per connection) that enables timeout once all writes are completed. Resets the state if the connection (and therefore pipeline's timer handler) is reused.

          Show
          jeagles Jonathan Eagles added a comment - Jason Lowe , added state to the timeout handler (one per connection) that enables timeout once all writes are completed. Resets the state if the connection (and therefore pipeline's timer handler) is reused.
          Hide
          jlowe Jason Lowe added a comment -

          Thanks for the patch!

          I'm wondering about handling the idle timeout. I'm worried about the case where disks are busy or slow. If it takes us longer than the idle timeout to respond to a request then I think the idle handler will close the connection mid-request which doesn't seem appropriate. Unless I'm missing something, the idle handler needs to check if we're in the middle of servicing a request or if we're waiting for the next request and only close for the latter.

          Show
          jlowe Jason Lowe added a comment - Thanks for the patch! I'm wondering about handling the idle timeout. I'm worried about the case where disks are busy or slow. If it takes us longer than the idle timeout to respond to a request then I think the idle handler will close the connection mid-request which doesn't seem appropriate. Unless I'm missing something, the idle handler needs to check if we're in the middle of servicing a request or if we're waiting for the next request and only close for the latter.
          Hide
          jeagles Jonathan Eagles added a comment -

          Jason Lowe, can you have a look at this patch to see if is ready to commit? Rajesh Balamohan and I are happy so far with the current patch but would like a second set of eyes.

          Show
          jeagles Jonathan Eagles added a comment - Jason Lowe , can you have a look at this patch to see if is ready to commit? Rajesh Balamohan and I are happy so far with the current patch but would like a second set of eyes.
          Hide
          rajesh.balamohan Rajesh Balamohan added a comment -

          Thanks for sharing the latest patch Jonathan Eagles. .4 patch lgtm.

          Show
          rajesh.balamohan Rajesh Balamohan added a comment - Thanks for sharing the latest patch Jonathan Eagles . .4 patch lgtm.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 23m 31s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 14m 58s trunk passed
          +1 compile 0m 16s trunk passed
          +1 checkstyle 0m 12s trunk passed
          +1 mvnsite 0m 16s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 23s trunk passed
          +1 javadoc 0m 12s trunk passed
          +1 mvninstall 0m 13s the patch passed
          +1 compile 0m 12s the patch passed
          +1 javac 0m 12s the patch passed
          +1 checkstyle 0m 8s the patch passed
          +1 mvnsite 0m 14s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 27s the patch passed
          +1 javadoc 0m 9s the patch passed
          +1 unit 0m 18s hadoop-mapreduce-client-shuffle in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          42m 52s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854983/MAPREDUCE-6850.4.patch
          JIRA Issue MAPREDUCE-6850
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 1173c58d808e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5f5b031
          Default Java 1.8.0_121
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6907/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6907/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 23m 31s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 14m 58s trunk passed +1 compile 0m 16s trunk passed +1 checkstyle 0m 12s trunk passed +1 mvnsite 0m 16s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 23s trunk passed +1 javadoc 0m 12s trunk passed +1 mvninstall 0m 13s the patch passed +1 compile 0m 12s the patch passed +1 javac 0m 12s the patch passed +1 checkstyle 0m 8s the patch passed +1 mvnsite 0m 14s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 27s the patch passed +1 javadoc 0m 9s the patch passed +1 unit 0m 18s hadoop-mapreduce-client-shuffle in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 42m 52s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854983/MAPREDUCE-6850.4.patch JIRA Issue MAPREDUCE-6850 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 1173c58d808e 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5f5b031 Default Java 1.8.0_121 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6907/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6907/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jeagles Jonathan Eagles added a comment -

          Rajesh Balamohan, posted v4 of the patch to address the extra Timer variable. Please review the latest version of the patch when you get a chance.

          Show
          jeagles Jonathan Eagles added a comment - Rajesh Balamohan , posted v4 of the patch to address the extra Timer variable. Please review the latest version of the patch when you get a chance.
          Hide
          rajesh.balamohan Rajesh Balamohan added a comment - - edited

          I checked in small multi-node cluster with the patch. Attaching the tcpdump screenshots for reference. Patch works fine with keep-alive enabled and connections are being reused, where mapOutputs are retrieved using same connection. Attachment "With_Patch.png" shows the TCP stream, where multiple mapOutput being fetched from same connection.

          One very minor comment in the patch. timer variable in HttpPipelineFactory may not be needed.

          In MAPREDUCE-5787, Keepalive parameter checks were there till https://issues.apache.org/jira/secure/attachment/12634984/MAPREDUCE-5787-2.4.0-v3.patch as follows.

          if (!keepAlive && !keepAliveParam) {
            lastMap.addListener(ChannelFutureListener.CLOSE);
          }
          

          However, during refactoring it got missed out in subsequent patches in the same JIRA. That caused this problem. However, It would have relied on client to close the connection. I.e it was the responsibility of the client (JDK's internal http client) to terminate the connection after keep-alive timeout. Current patch proposed in this JIRA addresses that scenario as well, where in it would automatically close the connection if timeout exceeds the threshold provided in server side.

          Show
          rajesh.balamohan Rajesh Balamohan added a comment - - edited I checked in small multi-node cluster with the patch. Attaching the tcpdump screenshots for reference. Patch works fine with keep-alive enabled and connections are being reused, where mapOutputs are retrieved using same connection. Attachment "With_Patch.png" shows the TCP stream, where multiple mapOutput being fetched from same connection. One very minor comment in the patch. timer variable in HttpPipelineFactory may not be needed. In MAPREDUCE-5787 , Keepalive parameter checks were there till https://issues.apache.org/jira/secure/attachment/12634984/MAPREDUCE-5787-2.4.0-v3.patch as follows. if (!keepAlive && !keepAliveParam) { lastMap.addListener(ChannelFutureListener.CLOSE); } However, during refactoring it got missed out in subsequent patches in the same JIRA. That caused this problem. However, It would have relied on client to close the connection. I.e it was the responsibility of the client (JDK's internal http client) to terminate the connection after keep-alive timeout. Current patch proposed in this JIRA addresses that scenario as well, where in it would automatically close the connection if timeout exceeds the threshold provided in server side.
          Hide
          rajesh.balamohan Rajesh Balamohan added a comment -

          Patch looks good to me. I would need more time for checking in cluster. Got DFSOutputStream timeout exception in the cluster I was trying out (which is not related to this jira)

          Show
          rajesh.balamohan Rajesh Balamohan added a comment - Patch looks good to me. I would need more time for checking in cluster. Got DFSOutputStream timeout exception in the cluster I was trying out (which is not related to this jira)
          Hide
          rajesh.balamohan Rajesh Balamohan added a comment -

          Thanks for the patch Jonathan Eagles. I am getting a small cluster today/tomorrow. I will check the patch and will update.

          Show
          rajesh.balamohan Rajesh Balamohan added a comment - Thanks for the patch Jonathan Eagles . I am getting a small cluster today/tomorrow. I will check the patch and will update.
          Hide
          jeagles Jonathan Eagles added a comment -

          Rajesh Balamohan, can you see if this patch fixes the keep-alive bug we were discussing and give a review?

          Show
          jeagles Jonathan Eagles added a comment - Rajesh Balamohan , can you see if this patch fixes the keep-alive bug we were discussing and give a review?
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 13m 14s trunk passed
          +1 compile 0m 16s trunk passed
          +1 checkstyle 0m 11s trunk passed
          +1 mvnsite 0m 16s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 24s trunk passed
          +1 javadoc 0m 12s trunk passed
          +1 mvninstall 0m 15s the patch passed
          +1 compile 0m 13s the patch passed
          +1 javac 0m 13s the patch passed
          +1 checkstyle 0m 9s the patch passed
          +1 mvnsite 0m 14s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 28s the patch passed
          +1 javadoc 0m 10s the patch passed
          +1 unit 0m 18s hadoop-mapreduce-client-shuffle in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          18m 1s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854367/MAPREDUCE-6850.3.patch
          JIRA Issue MAPREDUCE-6850
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0b7379c27af3 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 132f758
          Default Java 1.8.0_121
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6903/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6903/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 14s trunk passed +1 compile 0m 16s trunk passed +1 checkstyle 0m 11s trunk passed +1 mvnsite 0m 16s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 24s trunk passed +1 javadoc 0m 12s trunk passed +1 mvninstall 0m 15s the patch passed +1 compile 0m 13s the patch passed +1 javac 0m 13s the patch passed +1 checkstyle 0m 9s the patch passed +1 mvnsite 0m 14s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 28s the patch passed +1 javadoc 0m 10s the patch passed +1 unit 0m 18s hadoop-mapreduce-client-shuffle in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 18m 1s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854367/MAPREDUCE-6850.3.patch JIRA Issue MAPREDUCE-6850 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0b7379c27af3 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 132f758 Default Java 1.8.0_121 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6903/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6903/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 15m 10s trunk passed
          +1 compile 0m 17s trunk passed
          +1 checkstyle 0m 13s trunk passed
          +1 mvnsite 0m 18s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 0m 27s trunk passed
          +1 javadoc 0m 16s trunk passed
          +1 mvninstall 0m 16s the patch passed
          +1 compile 0m 17s the patch passed
          +1 javac 0m 17s the patch passed
          +1 checkstyle 0m 11s the patch passed
          +1 mvnsite 0m 17s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 0m 35s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 0m 12s the patch passed
          +1 unit 0m 19s hadoop-mapreduce-client-shuffle in the patch passed.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          20m 44s



          Reason Tests
          FindBugs module:hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
            Should org.apache.hadoop.mapred.ShuffleHandler$TimeoutHandler be a static inner class? At ShuffleHandler.java:inner class? At ShuffleHandler.java:[lines 781-787]



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854349/MAPREDUCE-6850.2.patch
          JIRA Issue MAPREDUCE-6850
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e3b8b65d8641 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 132f758
          Default Java 1.8.0_121
          findbugs v3.0.0
          findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6902/artifact/patchprocess/new-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.html
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6902/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6902/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 15m 10s trunk passed +1 compile 0m 17s trunk passed +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 18s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 27s trunk passed +1 javadoc 0m 16s trunk passed +1 mvninstall 0m 16s the patch passed +1 compile 0m 17s the patch passed +1 javac 0m 17s the patch passed +1 checkstyle 0m 11s the patch passed +1 mvnsite 0m 17s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 0m 35s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 0m 12s the patch passed +1 unit 0m 19s hadoop-mapreduce-client-shuffle in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 20m 44s Reason Tests FindBugs module:hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle   Should org.apache.hadoop.mapred.ShuffleHandler$TimeoutHandler be a static inner class? At ShuffleHandler.java:inner class? At ShuffleHandler.java: [lines 781-787] Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12854349/MAPREDUCE-6850.2.patch JIRA Issue MAPREDUCE-6850 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e3b8b65d8641 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 132f758 Default Java 1.8.0_121 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6902/artifact/patchprocess/new-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-shuffle.html Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6902/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-shuffle Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6902/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jeagles Jonathan Eagles added a comment -

          Details regarding network trace showing keep-alive connections not being reused can be found in TEZ-3633.

          Show
          jeagles Jonathan Eagles added a comment - Details regarding network trace showing keep-alive connections not being reused can be found in TEZ-3633 .

            People

            • Assignee:
              jeagles Jonathan Eagles
              Reporter:
              jeagles Jonathan Eagles
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development