Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13537

Support external calls in the RPC call queue

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: ipc
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Leveraging HADOOP-13465 will allow non-rpc calls to be added to the call queue. This is intended to support routing webhdfs calls through the call queue to provide a unified and protocol-independent QoS.

      1. HADOOP-13537-2.patch
        12 kB
        Rushabh S Shah
      2. HADOOP-13537-1.patch
        11 kB
        Rushabh S Shah
      3. HADOOP-13537.patch
        11 kB
        Daryn Sharp

        Issue Links

          Activity

          Hide
          shahrs87 Rushabh S Shah added a comment -

          Thanks Kihwal Lee for reviewing and committing.

          Show
          shahrs87 Rushabh S Shah added a comment - Thanks Kihwal Lee for reviewing and committing.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10515 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10515/)
          HADOOP-13537. Support external calls in the RPC call queue. Contributed (kihwal: rev 236ac773c964fa21d6d5f1496023cd61818dd3b1)

          • (edit) hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
          • (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ExternalCall.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10515 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10515/ ) HADOOP-13537 . Support external calls in the RPC call queue. Contributed (kihwal: rev 236ac773c964fa21d6d5f1496023cd61818dd3b1) (edit) hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ExternalCall.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java
          Hide
          kihwal Kihwal Lee added a comment -

          I've committed this to trunk, branch-2 and branch-2.8. Thanks for the original patch, Daryn. Big thanks to Rushabh for the rework.

          Show
          kihwal Kihwal Lee added a comment - I've committed this to trunk, branch-2 and branch-2.8. Thanks for the original patch, Daryn. Big thanks to Rushabh for the rework.
          Hide
          kihwal Kihwal Lee added a comment -

          +1 The latest patch looks good. This still does not change the existing behavior.

          Show
          kihwal Kihwal Lee added a comment - +1 The latest patch looks good. This still does not change the existing behavior.
          Hide
          shahrs87 Rushabh S Shah added a comment -
           Failed junit tests:
                  hadoop.ha.TestZKFailoverController
            	hadoop.net.TestDNS 
          

          Both the failed tests ran fine on my machine.

          Show
          shahrs87 Rushabh S Shah added a comment - Failed junit tests: hadoop.ha.TestZKFailoverController hadoop.net.TestDNS Both the failed tests ran fine on my machine.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 2s trunk passed
          +1 compile 8m 31s trunk passed
          +1 checkstyle 0m 29s trunk passed
          +1 mvnsite 1m 7s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 32s trunk passed
          +1 javadoc 0m 48s trunk passed
          +1 mvninstall 0m 44s the patch passed
          +1 compile 8m 4s the patch passed
          +1 javac 8m 4s the patch passed
          -0 checkstyle 0m 25s hadoop-common-project/hadoop-common: The patch generated 1 new + 223 unchanged - 0 fixed = 224 total (was 223)
          +1 mvnsite 1m 1s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 1m 39s the patch passed
          +1 javadoc 0m 48s the patch passed
          -1 unit 7m 55s hadoop-common in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          43m 40s



          Reason Tests
          Failed junit tests hadoop.ha.TestZKFailoverController
            hadoop.net.TestDNS



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13537
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830580/HADOOP-13537-2.patch
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0ffd510b215b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2acfb1e
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10618/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10618/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10618/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10618/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 2s trunk passed +1 compile 8m 31s trunk passed +1 checkstyle 0m 29s trunk passed +1 mvnsite 1m 7s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 32s trunk passed +1 javadoc 0m 48s trunk passed +1 mvninstall 0m 44s the patch passed +1 compile 8m 4s the patch passed +1 javac 8m 4s the patch passed -0 checkstyle 0m 25s hadoop-common-project/hadoop-common: The patch generated 1 new + 223 unchanged - 0 fixed = 224 total (was 223) +1 mvnsite 1m 1s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 39s the patch passed +1 javadoc 0m 48s the patch passed -1 unit 7m 55s hadoop-common in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 43m 40s Reason Tests Failed junit tests hadoop.ha.TestZKFailoverController   hadoop.net.TestDNS Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13537 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830580/HADOOP-13537-2.patch Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0ffd510b215b 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2acfb1e Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10618/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10618/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10618/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10618/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Attaching a new patch.

          Show
          shahrs87 Rushabh S Shah added a comment - Attaching a new patch.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Canceling patch to address checkstyle and findbugs warning.

          Show
          shahrs87 Rushabh S Shah added a comment - Canceling patch to address checkstyle and findbugs warning.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 35s trunk passed
          +1 compile 8m 22s trunk passed
          +1 checkstyle 0m 27s trunk passed
          +1 mvnsite 1m 2s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 35s trunk passed
          +1 javadoc 0m 50s trunk passed
          +1 mvninstall 0m 43s the patch passed
          +1 compile 8m 11s the patch passed
          +1 javac 8m 11s the patch passed
          -0 checkstyle 0m 28s hadoop-common-project/hadoop-common: The patch generated 17 new + 223 unchanged - 0 fixed = 240 total (was 223)
          +1 mvnsite 1m 3s the patch passed
          +1 mvneclipse 0m 15s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 51s hadoop-common-project/hadoop-common generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 javadoc 0m 54s the patch passed
          +1 unit 8m 48s hadoop-common in the patch passed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          45m 28s



          Reason Tests
          FindBugs module:hadoop-common-project/hadoop-common
            Synchronization performed on java.util.concurrent.atomic.AtomicBoolean in org.apache.hadoop.ipc.ExternalCall.doResponse(Throwable) At ExternalCall.java:org.apache.hadoop.ipc.ExternalCall.doResponse(Throwable) At ExternalCall.java:[line 85]
            Synchronization performed on java.util.concurrent.atomic.AtomicBoolean in org.apache.hadoop.ipc.ExternalCall.waitForCompletion() At ExternalCall.java:org.apache.hadoop.ipc.ExternalCall.waitForCompletion() At ExternalCall.java:[line 54]



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HADOOP-13537
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830360/HADOOP-13537-1.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a4a46565f3b1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4815d02
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10595/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10595/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10595/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10595/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 35s trunk passed +1 compile 8m 22s trunk passed +1 checkstyle 0m 27s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 35s trunk passed +1 javadoc 0m 50s trunk passed +1 mvninstall 0m 43s the patch passed +1 compile 8m 11s the patch passed +1 javac 8m 11s the patch passed -0 checkstyle 0m 28s hadoop-common-project/hadoop-common: The patch generated 17 new + 223 unchanged - 0 fixed = 240 total (was 223) +1 mvnsite 1m 3s the patch passed +1 mvneclipse 0m 15s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 51s hadoop-common-project/hadoop-common generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 0m 54s the patch passed +1 unit 8m 48s hadoop-common in the patch passed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 45m 28s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   Synchronization performed on java.util.concurrent.atomic.AtomicBoolean in org.apache.hadoop.ipc.ExternalCall.doResponse(Throwable) At ExternalCall.java:org.apache.hadoop.ipc.ExternalCall.doResponse(Throwable) At ExternalCall.java: [line 85]   Synchronization performed on java.util.concurrent.atomic.AtomicBoolean in org.apache.hadoop.ipc.ExternalCall.waitForCompletion() At ExternalCall.java:org.apache.hadoop.ipc.ExternalCall.waitForCompletion() At ExternalCall.java: [line 54] Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HADOOP-13537 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830360/HADOOP-13537-1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a4a46565f3b1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4815d02 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/10595/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt findbugs https://builds.apache.org/job/PreCommit-HADOOP-Build/10595/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10595/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10595/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Attaching a new patch.
          Rebased to all the latest changes.

          Show
          shahrs87 Rushabh S Shah added a comment - Attaching a new patch. Rebased to all the latest changes.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Current patch doesn't apply cleanly.
          Will submit a cleaner patch in sometime.

          Show
          shahrs87 Rushabh S Shah added a comment - Current patch doesn't apply cleanly. Will submit a cleaner patch in sometime.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 4s HADOOP-13537 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue HADOOP-13537
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825129/HADOOP-13537.patch
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10444/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 4s HADOOP-13537 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HADOOP-13537 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825129/HADOOP-13537.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10444/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          kihwal Kihwal Lee added a comment -

          Submitted...

          Show
          kihwal Kihwal Lee added a comment - Submitted...

            People

            • Assignee:
              daryn Daryn Sharp
              Reporter:
              daryn Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development