Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10789

Route webhdfs through the RPC call queue

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha2
    • Component/s: ipc, webhdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Webhdfs is extremely expensive under load and is not subject to the QoS benefits of the RPC call queue. HADOOP-13537 provides the basis for routing webhdfs through the call queue to provide unified QoS.

      1. HDFS-10789.patch
        21 kB
        Daryn Sharp
      2. HDFS-10789-1.patch
        21 kB
        Rushabh S Shah
      3. HDFS-10789-2.patch
        26 kB
        Rushabh S Shah
      4. HDFS-10789-3.patch
        26 kB
        Rushabh S Shah
      5. HDFS-10789-4.patch
        27 kB
        Rushabh S Shah
      6. HDFS-10789-5.patch
        27 kB
        Rushabh S Shah

        Issue Links

          Activity

          Hide
          kihwal Kihwal Lee added a comment -

          What happens when the call queue is full and backoff is on? What does the http client get and what does WebHdfsFilesSystem do in terms of retry? Does the connection go away or can they still pile up when the webhdfs load is high?

          Show
          kihwal Kihwal Lee added a comment - What happens when the call queue is full and backoff is on? What does the http client get and what does WebHdfsFilesSystem do in terms of retry? Does the connection go away or can they still pile up when the webhdfs load is high?
          Hide
          daryn Daryn Sharp added a comment -

          The current situation is all the jetty handler threads end up competing unfairly with the ipc handlers. The unregulated competition easily causes callq overflow and rpc calls are rejected. The jetty acceptor is still busy accepting an unlimited number of connections even while all the jetty handlers are blocked. A full GC is often inevitable when hit with an abusive surge.

          With this patch, the callq can provide fairness. Also assuming there's at least one jetty handler not blocked in the namesystem, new requests will be rejected and their connections will be closed.

          Show
          daryn Daryn Sharp added a comment - The current situation is all the jetty handler threads end up competing unfairly with the ipc handlers. The unregulated competition easily causes callq overflow and rpc calls are rejected. The jetty acceptor is still busy accepting an unlimited number of connections even while all the jetty handlers are blocked. A full GC is often inevitable when hit with an abusive surge. With this patch, the callq can provide fairness. Also assuming there's at least one jetty handler not blocked in the namesystem, new requests will be rejected and their connections will be closed.
          Hide
          kihwal Kihwal Lee added a comment -

          The patch does not apply anymore. Please rebase/update.

          Show
          kihwal Kihwal Lee added a comment - The patch does not apply anymore. Please rebase/update.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Attaching rebased patch.

          Show
          shahrs87 Rushabh S Shah added a comment - Attaching rebased patch.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks much for working on this Daryn Sharp Kihwal Lee and Rushabh S Shah.

          One feature we want is for WebHDFS calls to be restricted under the RPC ACLs (e.g. revoking a certain user's permission to issue certain WebHDFS calls). I haven't look into much detail of the patch. Any quick answer on this? Thanks!

          Show
          zhz Zhe Zhang added a comment - Thanks much for working on this Daryn Sharp Kihwal Lee and Rushabh S Shah . One feature we want is for WebHDFS calls to be restricted under the RPC ACLs (e.g. revoking a certain user's permission to issue certain WebHDFS calls). I haven't look into much detail of the patch. Any quick answer on this? Thanks!
          Hide
          daryn Daryn Sharp added a comment -

          The ACL authorization is currently only checked when processing the IPC connection context as part of the connection lifecycle prior to reading calls. Webhdfs has no connection context. I think applying common ACLs to webhdfs would be a useful feature but as a separate jira. Most likely it should be a change in the webhdfs servlet.

          Show
          daryn Daryn Sharp added a comment - The ACL authorization is currently only checked when processing the IPC connection context as part of the connection lifecycle prior to reading calls. Webhdfs has no connection context. I think applying common ACLs to webhdfs would be a useful feature but as a separate jira. Most likely it should be a change in the webhdfs servlet.
          Hide
          kihwal Kihwal Lee added a comment -

          The RPC ACL controls at protocol level, bit it looks like what you want is method level restrictions. It will best to open a new jira and present use cases to facilitate further discussions.

          Show
          kihwal Kihwal Lee added a comment - The RPC ACL controls at protocol level, bit it looks like what you want is method level restrictions. It will best to open a new jira and present use cases to facilitate further discussions.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 58s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 37s trunk passed
          +1 mvnsite 0m 50s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 42s trunk passed
          +1 javadoc 0m 55s trunk passed
          +1 mvninstall 0m 48s the patch passed
          +1 compile 0m 43s the patch passed
          +1 javac 0m 43s the patch passed
          -0 checkstyle 0m 34s hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 917 unchanged - 5 fixed = 925 total (was 922)
          +1 mvnsite 0m 49s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 49s the patch passed
          +1 javadoc 0m 52s the patch passed
          -1 unit 60m 13s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          79m 52s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestNamenodeRetryCache
            hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
            hadoop.hdfs.TestDFSShell
            hadoop.tools.TestHdfsConfigFields
            hadoop.hdfs.web.TestWebHDFSXAttr



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10789
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831389/HDFS-10789-1.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux f561fce8f501 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ef7f06f
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16997/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16997/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16997/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16997/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 58s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 37s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 55s trunk passed +1 mvninstall 0m 48s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed -0 checkstyle 0m 34s hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 917 unchanged - 5 fixed = 925 total (was 922) +1 mvnsite 0m 49s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 52s the patch passed -1 unit 60m 13s hadoop-hdfs in the patch failed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 79m 52s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNamenodeRetryCache   hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.TestDFSShell   hadoop.tools.TestHdfsConfigFields   hadoop.hdfs.web.TestWebHDFSXAttr Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10789 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831389/HDFS-10789-1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux f561fce8f501 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ef7f06f Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16997/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16997/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16997/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16997/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 18s Maven dependency ordering for branch
          +1 mvninstall 7m 7s trunk passed
          +1 compile 6m 46s trunk passed
          +1 checkstyle 1m 38s trunk passed
          +1 mvnsite 1m 47s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 2m 56s trunk passed
          +1 javadoc 1m 20s trunk passed
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 1m 19s the patch passed
          +1 compile 6m 42s the patch passed
          +1 javac 6m 42s the patch passed
          -0 checkstyle 1m 38s root: The patch generated 9 new + 971 unchanged - 5 fixed = 980 total (was 976)
          +1 mvnsite 1m 44s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 13s the patch passed
          +1 javadoc 1m 21s the patch passed
          +1 unit 7m 14s hadoop-common in the patch passed.
          -1 unit 57m 6s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 25s The patch does not generate ASF License warnings.
          104m 46s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10789
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832730/HDFS-10789-2.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 5276a9b6dac2 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2fb392a
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17106/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17106/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17106/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17106/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 18s Maven dependency ordering for branch +1 mvninstall 7m 7s trunk passed +1 compile 6m 46s trunk passed +1 checkstyle 1m 38s trunk passed +1 mvnsite 1m 47s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 2m 56s trunk passed +1 javadoc 1m 20s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 19s the patch passed +1 compile 6m 42s the patch passed +1 javac 6m 42s the patch passed -0 checkstyle 1m 38s root: The patch generated 9 new + 971 unchanged - 5 fixed = 980 total (was 976) +1 mvnsite 1m 44s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 13s the patch passed +1 javadoc 1m 21s the patch passed +1 unit 7m 14s hadoop-common in the patch passed. -1 unit 57m 6s hadoop-hdfs in the patch failed. +1 asflicense 0m 25s The patch does not generate ASF License warnings. 104m 46s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10789 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832730/HDFS-10789-2.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 5276a9b6dac2 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2fb392a Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17106/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17106/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17106/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17106/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Cancelling to address checkstyle issues.
          TestNameNodeMetadataConsistency#testGenerationStampInFuture test fails intermittently and is tracked by HDFS-10499.

          Show
          shahrs87 Rushabh S Shah added a comment - Cancelling to address checkstyle issues. TestNameNodeMetadataConsistency#testGenerationStampInFuture test fails intermittently and is tracked by HDFS-10499 .
          Hide
          shahrs87 Rushabh S Shah added a comment -

          forgot to address checkstyle in NameNodeRpcServer.java file

          Show
          shahrs87 Rushabh S Shah added a comment - forgot to address checkstyle in NameNodeRpcServer.java file
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 1m 33s Maven dependency ordering for branch
          +1 mvninstall 7m 27s trunk passed
          +1 compile 7m 44s trunk passed
          +1 checkstyle 1m 40s trunk passed
          +1 mvnsite 2m 0s trunk passed
          +1 mvneclipse 0m 28s trunk passed
          +1 findbugs 3m 11s trunk passed
          +1 javadoc 1m 25s trunk passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 1m 23s the patch passed
          +1 compile 6m 42s the patch passed
          +1 javac 6m 42s the patch passed
          -0 checkstyle 1m 38s root: The patch generated 2 new + 971 unchanged - 5 fixed = 973 total (was 976)
          +1 mvnsite 1m 43s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 3m 12s the patch passed
          +1 javadoc 1m 20s the patch passed
          -1 unit 21m 7s hadoop-common in the patch failed.
          +1 unit 58m 21s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          123m 4s



          Reason Tests
          Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10789
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832777/HDFS-10789-4.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux ee659129fbde 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8a09bf7
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17112/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17112/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17112/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17112/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 1m 33s Maven dependency ordering for branch +1 mvninstall 7m 27s trunk passed +1 compile 7m 44s trunk passed +1 checkstyle 1m 40s trunk passed +1 mvnsite 2m 0s trunk passed +1 mvneclipse 0m 28s trunk passed +1 findbugs 3m 11s trunk passed +1 javadoc 1m 25s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 23s the patch passed +1 compile 6m 42s the patch passed +1 javac 6m 42s the patch passed -0 checkstyle 1m 38s root: The patch generated 2 new + 971 unchanged - 5 fixed = 973 total (was 976) +1 mvnsite 1m 43s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 3m 12s the patch passed +1 javadoc 1m 20s the patch passed -1 unit 21m 7s hadoop-common in the patch failed. +1 unit 58m 21s hadoop-hdfs in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 123m 4s Reason Tests Timed out junit tests org.apache.hadoop.http.TestHttpServerLifecycle Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10789 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832777/HDFS-10789-4.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux ee659129fbde 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8a09bf7 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17112/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17112/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17112/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17112/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          I missed 2 checkstyle warnings again.

          Show
          shahrs87 Rushabh S Shah added a comment - I missed 2 checkstyle warnings again.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Attaching a patch fixing 2 checkstyle warnings.
          Hopefully the last patch.

          Show
          shahrs87 Rushabh S Shah added a comment - Attaching a patch fixing 2 checkstyle warnings. Hopefully the last patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 12s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 3 new or modified test files.
          0 mvndep 0m 14s Maven dependency ordering for branch
          +1 mvninstall 6m 32s trunk passed
          +1 compile 6m 46s trunk passed
          +1 checkstyle 1m 38s trunk passed
          +1 mvnsite 1m 48s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 2m 58s trunk passed
          +1 javadoc 1m 20s trunk passed
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 1m 20s the patch passed
          +1 compile 6m 43s the patch passed
          +1 javac 6m 43s the patch passed
          +1 checkstyle 1m 38s root: The patch generated 0 new + 971 unchanged - 5 fixed = 971 total (was 976)
          +1 mvnsite 1m 43s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 3m 14s the patch passed
          +1 javadoc 1m 20s the patch passed
          +1 unit 7m 59s hadoop-common in the patch passed.
          +1 unit 57m 41s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          105m 24s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-10789
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832925/HDFS-10789-5.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 83f74f2f68fe 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 6476934
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17122/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17122/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 3 new or modified test files. 0 mvndep 0m 14s Maven dependency ordering for branch +1 mvninstall 6m 32s trunk passed +1 compile 6m 46s trunk passed +1 checkstyle 1m 38s trunk passed +1 mvnsite 1m 48s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 2m 58s trunk passed +1 javadoc 1m 20s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 20s the patch passed +1 compile 6m 43s the patch passed +1 javac 6m 43s the patch passed +1 checkstyle 1m 38s root: The patch generated 0 new + 971 unchanged - 5 fixed = 971 total (was 976) +1 mvnsite 1m 43s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 14s the patch passed +1 javadoc 1m 20s the patch passed +1 unit 7m 59s hadoop-common in the patch passed. +1 unit 57m 41s hadoop-hdfs in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 105m 24s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-10789 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12832925/HDFS-10789-5.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 83f74f2f68fe 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6476934 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17122/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17122/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          kihwal Kihwal Lee added a comment -

          +1 looks good. We have been running this internally for quite some time (over 6 months).

          Show
          kihwal Kihwal Lee added a comment - +1 looks good. We have been running this internally for quite some time (over 6 months).
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10597 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10597/)
          HDFS-10789. Route webhdfs through the RPC call queue. Contributed by (kihwal: rev 85cd06f6636f295ad1f3bf2a90063f4714c9cca7)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ExternalCall.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/web/resources/TestWebHdfsDataLocality.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10597 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10597/ ) HDFS-10789 . Route webhdfs through the RPC call queue. Contributed by (kihwal: rev 85cd06f6636f295ad1f3bf2a90063f4714c9cca7) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNamenodeRetryCache.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ExternalCall.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestRPC.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/web/resources/TestWebHdfsDataLocality.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/web/resources/NamenodeWebHdfsMethods.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
          Hide
          kihwal Kihwal Lee added a comment -

          Committed this to trunk, branch-2 and branch-2.8.
          Thanks for providing the original patch, Daryn. Also thanks Rushabh for driving this to the end.

          Show
          kihwal Kihwal Lee added a comment - Committed this to trunk, branch-2 and branch-2.8. Thanks for providing the original patch, Daryn. Also thanks Rushabh for driving this to the end.
          Hide
          shahrs87 Rushabh S Shah added a comment -

          Thanks Kihwal Lee for the reviews and commit.

          Show
          shahrs87 Rushabh S Shah added a comment - Thanks Kihwal Lee for the reviews and commit.
          Hide
          kihwal Kihwal Lee added a comment -

          I remember we had to fix one bug after this feature. Filed HDFS-11000.

          Show
          kihwal Kihwal Lee added a comment - I remember we had to fix one bug after this feature. Filed HDFS-11000 .

            People

            • Assignee:
              daryn Daryn Sharp
              Reporter:
              daryn Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development