Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Will be useful to figure out when the NN is getting overloaded with more connections than it can handle

      1. HADOOP-14440.003.patch
        6 kB
        Eric Badger
      2. HADOOP-14440.002.patch
        4 kB
        Eric Badger
      3. HADOOP-14440.001.patch
        4 kB
        Eric Badger

        Issue Links

          Activity

          Hide
          ebadger Eric Badger added a comment -

          Uploading patch with new metric for dropped connections

          Show
          ebadger Eric Badger added a comment - Uploading patch with new metric for dropped connections
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 15m 16s trunk passed
          +1 compile 14m 24s trunk passed
          +1 checkstyle 0m 39s trunk passed
          +1 mvnsite 1m 7s trunk passed
          +1 mvneclipse 0m 20s trunk passed
          -1 findbugs 1m 29s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings.
          +1 javadoc 0m 51s trunk passed
          +1 mvninstall 0m 40s the patch passed
          +1 compile 13m 3s the patch passed
          +1 javac 13m 3s the patch passed
          -0 checkstyle 0m 39s hadoop-common-project/hadoop-common: The patch generated 1 new + 308 unchanged - 0 fixed = 309 total (was 308)
          +1 mvnsite 1m 11s the patch passed
          +1 mvneclipse 0m 18s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 53s the patch passed
          +1 unit 8m 50s hadoop-common in the patch passed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          64m 20s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14440
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12868996/HADOOP-14440.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e77349edd3d3 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 19482e0
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12361/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12361/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12361/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12361/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 15m 16s trunk passed +1 compile 14m 24s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 1m 7s trunk passed +1 mvneclipse 0m 20s trunk passed -1 findbugs 1m 29s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings. +1 javadoc 0m 51s trunk passed +1 mvninstall 0m 40s the patch passed +1 compile 13m 3s the patch passed +1 javac 13m 3s the patch passed -0 checkstyle 0m 39s hadoop-common-project/hadoop-common: The patch generated 1 new + 308 unchanged - 0 fixed = 309 total (was 308) +1 mvnsite 1m 11s the patch passed +1 mvneclipse 0m 18s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 53s the patch passed +1 unit 8m 50s hadoop-common in the patch passed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 64m 20s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14440 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12868996/HADOOP-14440.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e77349edd3d3 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 19482e0 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12361/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12361/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12361/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12361/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          nroberts Nathan Roberts added a comment -

          Thanks Eric. One minor nit, could name the getter droppedConnections() as getDroppedConnections().

          Show
          nroberts Nathan Roberts added a comment - Thanks Eric. One minor nit, could name the getter droppedConnections() as getDroppedConnections().
          Hide
          ebadger Eric Badger added a comment -

          Thanks, Nathan Roberts! Uploading new patch that fixes the naming nit

          Show
          ebadger Eric Badger added a comment - Thanks, Nathan Roberts ! Uploading new patch that fixes the naming nit
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 12m 15s trunk passed
          +1 compile 12m 19s trunk passed
          +1 checkstyle 0m 40s trunk passed
          +1 mvnsite 1m 4s trunk passed
          +1 mvneclipse 0m 17s trunk passed
          -1 findbugs 1m 22s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings.
          +1 javadoc 0m 53s trunk passed
          +1 mvninstall 0m 36s the patch passed
          +1 compile 11m 47s the patch passed
          +1 javac 11m 47s the patch passed
          -0 checkstyle 0m 33s hadoop-common-project/hadoop-common: The patch generated 1 new + 308 unchanged - 0 fixed = 309 total (was 308)
          +1 mvnsite 1m 0s the patch passed
          +1 mvneclipse 0m 23s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 32s the patch passed
          +1 javadoc 0m 51s the patch passed
          -1 unit 7m 2s hadoop-common in the patch failed.
          +1 asflicense 0m 36s The patch does not generate ASF License warnings.
          55m 29s



          Reason Tests
          Failed junit tests hadoop.security.TestRaceWhenRelogin



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14440
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869047/HADOOP-14440.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8f072bf420f1 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6672810
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12365/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12365/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12365/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12365/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12365/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 12m 15s trunk passed +1 compile 12m 19s trunk passed +1 checkstyle 0m 40s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 17s trunk passed -1 findbugs 1m 22s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings. +1 javadoc 0m 53s trunk passed +1 mvninstall 0m 36s the patch passed +1 compile 11m 47s the patch passed +1 javac 11m 47s the patch passed -0 checkstyle 0m 33s hadoop-common-project/hadoop-common: The patch generated 1 new + 308 unchanged - 0 fixed = 309 total (was 308) +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 23s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 32s the patch passed +1 javadoc 0m 51s the patch passed -1 unit 7m 2s hadoop-common in the patch failed. +1 asflicense 0m 36s The patch does not generate ASF License warnings. 55m 29s Reason Tests Failed junit tests hadoop.security.TestRaceWhenRelogin Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14440 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869047/HADOOP-14440.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8f072bf420f1 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6672810 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12365/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12365/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12365/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12365/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12365/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          Test failure is unrelated to the patch and possibly related to HADOOP-14078

          Show
          ebadger Eric Badger added a comment - Test failure is unrelated to the patch and possibly related to HADOOP-14078
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Eric Badger thanks for working on this.. Approach LGTM.

          i) Can you please update metrics.md also..?
          ii) how about adding seperate testcase for this,testMaxConnections can be refactored Or can you comment before assertion..?
          iii) Address the checkstyle issue to make jenkins happy..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Eric Badger thanks for working on this.. Approach LGTM. i) Can you please update metrics.md also..? ii) how about adding seperate testcase for this, testMaxConnections can be refactored Or can you comment before assertion..? iii) Address the checkstyle issue to make jenkins happy..
          Hide
          ebadger Eric Badger added a comment -

          Thanks for the review, Brahma Reddy Battula!

          Uploading patch with updated metrics.md, modified comments in the test case to make it more clear what is going on, and fixed the checkstyle.

          Show
          ebadger Eric Badger added a comment - Thanks for the review, Brahma Reddy Battula ! Uploading patch with updated metrics.md, modified comments in the test case to make it more clear what is going on, and fixed the checkstyle.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 51s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 15m 28s trunk passed
          +1 compile 15m 31s trunk passed
          +1 checkstyle 0m 39s trunk passed
          +1 mvnsite 1m 13s trunk passed
          +1 mvneclipse 0m 20s trunk passed
          -1 findbugs 1m 39s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings.
          +1 javadoc 0m 53s trunk passed
          +1 mvninstall 0m 49s the patch passed
          +1 compile 14m 39s the patch passed
          +1 javac 14m 39s the patch passed
          +1 checkstyle 0m 43s the patch passed
          +1 mvnsite 1m 6s the patch passed
          +1 mvneclipse 0m 21s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 35s the patch passed
          +1 javadoc 0m 49s the patch passed
          +1 unit 8m 42s hadoop-common in the patch passed.
          +1 asflicense 0m 40s The patch does not generate ASF License warnings.
          67m 59s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HADOOP-14440
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869305/HADOOP-14440.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 81335b68f5b2 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2a8fcf0
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12373/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12373/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12373/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 51s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 15m 28s trunk passed +1 compile 15m 31s trunk passed +1 checkstyle 0m 39s trunk passed +1 mvnsite 1m 13s trunk passed +1 mvneclipse 0m 20s trunk passed -1 findbugs 1m 39s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings. +1 javadoc 0m 53s trunk passed +1 mvninstall 0m 49s the patch passed +1 compile 14m 39s the patch passed +1 javac 14m 39s the patch passed +1 checkstyle 0m 43s the patch passed +1 mvnsite 1m 6s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 35s the patch passed +1 javadoc 0m 49s the patch passed +1 unit 8m 42s hadoop-common in the patch passed. +1 asflicense 0m 40s The patch does not generate ASF License warnings. 67m 59s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HADOOP-14440 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12869305/HADOOP-14440.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 81335b68f5b2 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2a8fcf0 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/12373/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12373/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12373/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Eric Badger thanks for updating the patch.. +1 on latest patch.. will commit this weekend unless there are no comments from somebody.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Eric Badger thanks for updating the patch.. +1 on latest patch.. will commit this weekend unless there are no comments from somebody.
          Hide
          ebadger Eric Badger added a comment -

          Hi Brahma Reddy Battula, looks like there are no other comments. Would you mind committing this?

          Show
          ebadger Eric Badger added a comment - Hi Brahma Reddy Battula , looks like there are no other comments. Would you mind committing this?
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Yes,sure..will commit shortly

          Show
          brahmareddy Brahma Reddy Battula added a comment - Yes,sure..will commit shortly
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Committed to trunk and branch-2..Eric Badger thanks for your contribution.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Committed to trunk and branch-2.. Eric Badger thanks for your contribution.
          Hide
          ebadger Eric Badger added a comment -
          Show
          ebadger Eric Badger added a comment - Thanks, Brahma Reddy Battula !
          Hide
          kihwal Kihwal Lee added a comment -

          Cherry-picked to branch-2.8.

          Show
          kihwal Kihwal Lee added a comment - Cherry-picked to branch-2.8.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11822 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11822/)
          HADOOP-14440. Add metrics for connections dropped. Contributed by Eric (brahma: rev abdd609e51a80388493417126c3bc9b1badc0ac1)

          • (edit) hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11822 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11822/ ) HADOOP-14440 . Add metrics for connections dropped. Contributed by Eric (brahma: rev abdd609e51a80388493417126c3bc9b1badc0ac1) (edit) hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestIPC.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/metrics/RpcMetrics.java
          Hide
          zhz Zhe Zhang added a comment -

          Thanks for the work Eric Badger. I think this is a good improvement for 2.7.4; just backported to branch-2.7.

          Show
          zhz Zhe Zhang added a comment - Thanks for the work Eric Badger . I think this is a good improvement for 2.7.4; just backported to branch-2.7.

            People

            • Assignee:
              ebadger Eric Badger
              Reporter:
              ebadger Eric Badger
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development