Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11538

Move ClientProtocol HA proxies into hadoop-hdfs-client

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 2.8.0, 3.0.0-alpha1
    • Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
    • Component/s: hdfs-client
    • Labels:
      None

      Description

      Follow-up for HDFS-11431. We should move this missing class over rather than pulling in the whole hadoop-hdfs dependency.

      1. HDFS-11538-branch-2.001.patch
        53 kB
        Huafeng Wang
      2. HDFS-11538.003.patch
        55 kB
        Huafeng Wang
      3. HDFS-11538.002.patch
        53 kB
        Huafeng Wang
      4. HDFS-11538.001.patch
        37 kB
        Huafeng Wang

        Issue Links

          Activity

          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.
          Hide
          djp Junping Du added a comment -

          Cool. Thanks Andrew!

          Show
          djp Junping Du added a comment - Cool. Thanks Andrew!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11591 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11591/)
          HDFS-11538. Move ClientProtocol HA proxies into hadoop-hdfs-client. (wang: rev 9e0e430f18d45cfe125dda8d85916edddf79e8d6)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          • (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ConfiguredFailoverProxyProvider.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
          • (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ConfiguredFailoverProxyProvider.java
          • (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/IPFailoverProxyProvider.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HAUtilClient.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
          • (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
          • (delete) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRequestHedgingProxyProvider.java
          • (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ClientHAProxyFactory.java
          • (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/HAProxyFactory.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DfsServlet.java
          • (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/NameNodeHAProxyFactory.java
          • (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/IPFailoverProxyProvider.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java
          • (add) hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRequestHedgingProxyProvider.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11591 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11591/ ) HDFS-11538 . Move ClientProtocol HA proxies into hadoop-hdfs-client. (wang: rev 9e0e430f18d45cfe125dda8d85916edddf79e8d6) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ConfiguredFailoverProxyProvider.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ConfiguredFailoverProxyProvider.java (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/IPFailoverProxyProvider.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HAUtilClient.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java (delete) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRequestHedgingProxyProvider.java (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ClientHAProxyFactory.java (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/HAProxyFactory.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DfsServlet.java (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/NameNodeHAProxyFactory.java (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/IPFailoverProxyProvider.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java (add) hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRequestHedgingProxyProvider.java
          Hide
          andrew.wang Andrew Wang added a comment -

          Cool. I did the requisite JIRA stuff to make the release notes track properly, and also pulled in these other good looking fixes to make the 2.8.1 backport clean:

          d12a0a2 (HEAD -> refs/heads/branch-2.8) HDFS-11538. Move ClientProtocol HA proxies into hadoop-hdfs-client. Contributed by Huafeng Wang.
          eff4b2f HDFS-10683. Make class Token$PrivateToken private. Contributed by John Zhuge.
          7ad5b27 HDFS-9276. Failed to Update HDFS Delegation Token for long running application in HA mode. Contributed by Liangliang Gu and John Zhuge
          e216c15 HDFS-11395. RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode. Contributed by Nandakumar.
          ab673aa HDFS-11629. Revert "HDFS-11431. hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider."
          
          Show
          andrew.wang Andrew Wang added a comment - Cool. I did the requisite JIRA stuff to make the release notes track properly, and also pulled in these other good looking fixes to make the 2.8.1 backport clean: d12a0a2 (HEAD -> refs/heads/branch-2.8) HDFS-11538. Move ClientProtocol HA proxies into hadoop-hdfs-client. Contributed by Huafeng Wang. eff4b2f HDFS-10683. Make class Token$PrivateToken private. Contributed by John Zhuge. 7ad5b27 HDFS-9276. Failed to Update HDFS Delegation Token for long running application in HA mode. Contributed by Liangliang Gu and John Zhuge e216c15 HDFS-11395. RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode. Contributed by Nandakumar. ab673aa HDFS-11629. Revert "HDFS-11431. hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider."
          Hide
          djp Junping Du added a comment -

          Thanks Andrew Wang and all who work on this JIRA. +1 on get this fix in 2.8.1 and revert HDFS-11431.

          Show
          djp Junping Du added a comment - Thanks Andrew Wang and all who work on this JIRA. +1 on get this fix in 2.8.1 and revert HDFS-11431 .
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks again Huafeng. I've reverted HDFS-11431 from branch-2 and committed this one (HDFS-11538).

          Junping Du if you'd like this for 2.8.1, let us know. I think it's worthwhile. We'd need another JIRA to track the revert of HDFS-11431 though since 2.8.0 went out.

          Show
          andrew.wang Andrew Wang added a comment - Thanks again Huafeng. I've reverted HDFS-11431 from branch-2 and committed this one ( HDFS-11538 ). Junping Du if you'd like this for 2.8.1, let us know. I think it's worthwhile. We'd need another JIRA to track the revert of HDFS-11431 though since 2.8.0 went out.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 14m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 33s Maven dependency ordering for branch
          +1 mvninstall 7m 40s branch-2 passed
          +1 compile 1m 26s branch-2 passed with JDK v1.8.0_121
          +1 compile 1m 33s branch-2 passed with JDK v1.7.0_121
          +1 checkstyle 0m 45s branch-2 passed
          +1 mvnsite 1m 36s branch-2 passed
          +1 mvneclipse 0m 32s branch-2 passed
          +1 findbugs 4m 25s branch-2 passed
          +1 javadoc 1m 17s branch-2 passed with JDK v1.8.0_121
          +1 javadoc 2m 0s branch-2 passed with JDK v1.7.0_121
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 1m 18s the patch passed
          +1 compile 1m 14s the patch passed with JDK v1.8.0_121
          +1 javac 1m 14s the patch passed
          +1 compile 1m 22s the patch passed with JDK v1.7.0_121
          +1 javac 1m 22s the patch passed
          -0 checkstyle 0m 40s hadoop-hdfs-project: The patch generated 20 new + 629 unchanged - 26 fixed = 649 total (was 655)
          +1 mvnsite 1m 23s the patch passed
          +1 mvneclipse 0m 25s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 4m 4s the patch passed
          +1 javadoc 1m 9s the patch passed with JDK v1.8.0_121
          +1 javadoc 1m 56s the patch passed with JDK v1.7.0_121
          +1 unit 1m 17s hadoop-hdfs-client in the patch passed with JDK v1.7.0_121.
          -1 unit 48m 38s hadoop-hdfs in the patch failed with JDK v1.7.0_121.
          +1 asflicense 0m 20s The patch does not generate ASF License warnings.
          156m 3s



          Reason Tests
          JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Issue HDFS-11538
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862050/HDFS-11538-branch-2.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 02d714bf4516 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 4cdda4b
          Default Java 1.7.0_121
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18986/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18986/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt
          JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18986/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18986/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 14m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 33s Maven dependency ordering for branch +1 mvninstall 7m 40s branch-2 passed +1 compile 1m 26s branch-2 passed with JDK v1.8.0_121 +1 compile 1m 33s branch-2 passed with JDK v1.7.0_121 +1 checkstyle 0m 45s branch-2 passed +1 mvnsite 1m 36s branch-2 passed +1 mvneclipse 0m 32s branch-2 passed +1 findbugs 4m 25s branch-2 passed +1 javadoc 1m 17s branch-2 passed with JDK v1.8.0_121 +1 javadoc 2m 0s branch-2 passed with JDK v1.7.0_121 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 18s the patch passed +1 compile 1m 14s the patch passed with JDK v1.8.0_121 +1 javac 1m 14s the patch passed +1 compile 1m 22s the patch passed with JDK v1.7.0_121 +1 javac 1m 22s the patch passed -0 checkstyle 0m 40s hadoop-hdfs-project: The patch generated 20 new + 629 unchanged - 26 fixed = 649 total (was 655) +1 mvnsite 1m 23s the patch passed +1 mvneclipse 0m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 4m 4s the patch passed +1 javadoc 1m 9s the patch passed with JDK v1.8.0_121 +1 javadoc 1m 56s the patch passed with JDK v1.7.0_121 +1 unit 1m 17s hadoop-hdfs-client in the patch passed with JDK v1.7.0_121. -1 unit 48m 38s hadoop-hdfs in the patch failed with JDK v1.7.0_121. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 156m 3s Reason Tests JDK v1.7.0_121 Failed junit tests hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Issue HDFS-11538 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12862050/HDFS-11538-branch-2.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 02d714bf4516 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 4cdda4b Default Java 1.7.0_121 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_121 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18986/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18986/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_121.txt JDK v1.7.0_121 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18986/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18986/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          HuafengWang Huafeng Wang added a comment -

          Got it, I'm preparing the patch.

          Show
          HuafengWang Huafeng Wang added a comment - Got it, I'm preparing the patch.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11528 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11528/)
          HDFS-11538. Move ClientProtocol HA proxies into hadoop-hdfs-client. (wang: rev 9e0e430f18d45cfe125dda8d85916edddf79e8d6)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java
          • (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
          • (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ConfiguredFailoverProxyProvider.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
          • (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ConfiguredFailoverProxyProvider.java
          • (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DfsServlet.java
          • (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/NameNodeHAProxyFactory.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HAUtilClient.java
          • (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/HAProxyFactory.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java
          • (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ClientHAProxyFactory.java
          • (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/IPFailoverProxyProvider.java
          • (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/IPFailoverProxyProvider.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • (delete) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRequestHedgingProxyProvider.java
          • (add) hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRequestHedgingProxyProvider.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11528 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11528/ ) HDFS-11538 . Move ClientProtocol HA proxies into hadoop-hdfs-client. (wang: rev 9e0e430f18d45cfe125dda8d85916edddf79e8d6) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ConfiguredFailoverProxyProvider.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ConfiguredFailoverProxyProvider.java (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/RequestHedgingProxyProvider.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/DfsServlet.java (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/NameNodeHAProxyFactory.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientFailover.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HAUtilClient.java (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/HAProxyFactory.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/ClientHAProxyFactory.java (add) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/IPFailoverProxyProvider.java (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/IPFailoverProxyProvider.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java (delete) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRequestHedgingProxyProvider.java (add) hadoop-hdfs-project/hadoop-hdfs-client/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestRequestHedgingProxyProvider.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for the contribution Huafeng Wang, I've committed this to trunk. Do you mind also preparing a patch for branch-2? Leaving this JIRA open for now.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for the contribution Huafeng Wang , I've committed this to trunk. Do you mind also preparing a patch for branch-2? Leaving this JIRA open for now.
          Hide
          HuafengWang Huafeng Wang added a comment -

          Hi Andrew, I just uploaded a new patch which mainly fix the whitespace issue. I think the UT failure is irrelevant and it passed on my local machine.

          Show
          HuafengWang Huafeng Wang added a comment - Hi Andrew, I just uploaded a new patch which mainly fix the whitespace issue. I think the UT failure is irrelevant and it passed on my local machine.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 24s Maven dependency ordering for branch
          +1 mvninstall 12m 51s trunk passed
          +1 compile 1m 22s trunk passed
          +1 checkstyle 0m 48s trunk passed
          +1 mvnsite 1m 24s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 10s trunk passed
          +1 javadoc 0m 59s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 16s the patch passed
          +1 compile 1m 20s the patch passed
          +1 javac 1m 20s the patch passed
          -0 checkstyle 0m 45s hadoop-hdfs-project: The patch generated 22 new + 652 unchanged - 29 fixed = 674 total (was 681)
          +1 mvnsite 1m 21s the patch passed
          +1 mvneclipse 0m 20s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 21s the patch passed
          +1 javadoc 0m 56s the patch passed
          +1 unit 1m 8s hadoop-hdfs-client in the patch passed.
          -1 unit 68m 47s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          102m 45s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11538
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861997/HDFS-11538.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d321578533ec 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2d5c09b
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18979/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18979/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18979/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18979/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 24s Maven dependency ordering for branch +1 mvninstall 12m 51s trunk passed +1 compile 1m 22s trunk passed +1 checkstyle 0m 48s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 10s trunk passed +1 javadoc 0m 59s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 16s the patch passed +1 compile 1m 20s the patch passed +1 javac 1m 20s the patch passed -0 checkstyle 0m 45s hadoop-hdfs-project: The patch generated 22 new + 652 unchanged - 29 fixed = 674 total (was 681) +1 mvnsite 1m 21s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 21s the patch passed +1 javadoc 0m 56s the patch passed +1 unit 1m 8s hadoop-hdfs-client in the patch passed. -1 unit 68m 47s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 102m 45s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDirectoryScanner Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11538 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861997/HDFS-11538.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d321578533ec 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2d5c09b Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18979/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18979/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18979/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18979/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          andrew.wang Andrew Wang added a comment -

          LGTM +1 pending the checkstyles and whitespace errors flagged by precommit. We don't need to fix all of them, but please make your best efforts.

          Thanks again for working on this Huafeng Wang!

          Show
          andrew.wang Andrew Wang added a comment - LGTM +1 pending the checkstyles and whitespace errors flagged by precommit. We don't need to fix all of them, but please make your best efforts. Thanks again for working on this Huafeng Wang !
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
          0 mvndep 0m 25s Maven dependency ordering for branch
          +1 mvninstall 12m 51s trunk passed
          +1 compile 1m 23s trunk passed
          +1 checkstyle 0m 49s trunk passed
          +1 mvnsite 1m 31s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 3m 9s trunk passed
          +1 javadoc 1m 1s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 17s the patch passed
          +1 compile 1m 19s the patch passed
          +1 javac 1m 19s the patch passed
          -0 checkstyle 0m 47s hadoop-hdfs-project: The patch generated 43 new + 653 unchanged - 30 fixed = 696 total (was 683)
          +1 mvnsite 1m 18s the patch passed
          +1 mvneclipse 0m 20s the patch passed
          -1 whitespace 0m 0s The patch has 12 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 3m 19s the patch passed
          +1 javadoc 0m 56s the patch passed
          +1 unit 1m 8s hadoop-hdfs-client in the patch passed.
          -1 unit 64m 12s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          98m 11s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11538
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861368/HDFS-11538.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 731180a892b8 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bf3fb58
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18919/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18919/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/18919/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18919/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18919/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 25s Maven dependency ordering for branch +1 mvninstall 12m 51s trunk passed +1 compile 1m 23s trunk passed +1 checkstyle 0m 49s trunk passed +1 mvnsite 1m 31s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 9s trunk passed +1 javadoc 1m 1s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 17s the patch passed +1 compile 1m 19s the patch passed +1 javac 1m 19s the patch passed -0 checkstyle 0m 47s hadoop-hdfs-project: The patch generated 43 new + 653 unchanged - 30 fixed = 696 total (was 683) +1 mvnsite 1m 18s the patch passed +1 mvneclipse 0m 20s the patch passed -1 whitespace 0m 0s The patch has 12 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 3m 19s the patch passed +1 javadoc 0m 56s the patch passed +1 unit 1m 8s hadoop-hdfs-client in the patch passed. -1 unit 64m 12s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 98m 11s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11538 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861368/HDFS-11538.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 731180a892b8 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bf3fb58 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18919/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18919/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/18919/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18919/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18919/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          HuafengWang Huafeng Wang added a comment -

          Thanks Sean Busbey for your advice. I have moved the related unit test into hadoop-hdfs-client module.

          And also thanks Andrew Wang for your comments, I have addressed these in the new patch. For the test problem, currently I moved the TestRequestHedgingProxyProvider into hadoop-hdfs-client module, and the test goes well, do you think it means that the failover proxies can be instantiated without hadoop-hdfs dependency?

          Show
          HuafengWang Huafeng Wang added a comment - Thanks Sean Busbey for your advice. I have moved the related unit test into hadoop-hdfs-client module. And also thanks Andrew Wang for your comments, I have addressed these in the new patch. For the test problem, currently I moved the TestRequestHedgingProxyProvider into hadoop-hdfs-client module, and the test goes well, do you think it means that the failover proxies can be instantiated without hadoop-hdfs dependency?
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for working on this Huafeng Wang this looks really, really good. I only have some minor comments:

          • Looking through the users of NamenodeProxies, we can remove the method in DfsServlet that calls it since it's unused.
          • NNProxiesClient#createFPP with a proxy factory can be made package private, no need for VisibleForTesting either.
          • Recommend rename HAProxyFactoryClient to ClientHAProxyFactory to unify naming with NameNodeHAProxyFactory
          • Don't need to fix the existing whitespace and checkstyle issues, but we should at least fix the unused imports from moving code around

          I would like to see some client unit tests that show we can instantiate the failover proxies without a hadoop-hdfs dependency. TestIsMethodSupported looks like a good place to start.

          Unfortunately, we heavily depend on miniclusters for testing, which makes splitting the unit test suite very challenging. Maybe this can be addressed by using the shaded minicluster (or lots of mocking), but I think it's too big a lift to pursue in this change. I still think it'd be a really great follow-on JIRA though.

          Show
          andrew.wang Andrew Wang added a comment - Thanks for working on this Huafeng Wang this looks really, really good. I only have some minor comments: Looking through the users of NamenodeProxies, we can remove the method in DfsServlet that calls it since it's unused. NNProxiesClient#createFPP with a proxy factory can be made package private, no need for VisibleForTesting either. Recommend rename HAProxyFactoryClient to ClientHAProxyFactory to unify naming with NameNodeHAProxyFactory Don't need to fix the existing whitespace and checkstyle issues, but we should at least fix the unused imports from moving code around I would like to see some client unit tests that show we can instantiate the failover proxies without a hadoop-hdfs dependency. TestIsMethodSupported looks like a good place to start. Unfortunately, we heavily depend on miniclusters for testing, which makes splitting the unit test suite very challenging. Maybe this can be addressed by using the shaded minicluster (or lots of mocking), but I think it's too big a lift to pursue in this change. I still think it'd be a really great follow-on JIRA though.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 28s Maven dependency ordering for branch
          +1 mvninstall 13m 7s trunk passed
          +1 compile 1m 22s trunk passed
          +1 checkstyle 0m 48s trunk passed
          +1 mvnsite 1m 24s trunk passed
          +1 mvneclipse 0m 26s trunk passed
          +1 findbugs 3m 8s trunk passed
          +1 javadoc 1m 0s trunk passed
          0 mvndep 0m 7s Maven dependency ordering for patch
          +1 mvninstall 1m 16s the patch passed
          +1 compile 1m 19s the patch passed
          +1 javac 1m 19s the patch passed
          -0 checkstyle 0m 46s hadoop-hdfs-project: The patch generated 43 new + 656 unchanged - 24 fixed = 699 total (was 680)
          +1 mvnsite 1m 19s the patch passed
          +1 mvneclipse 0m 20s the patch passed
          -1 whitespace 0m 0s The patch has 12 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          +1 findbugs 3m 20s the patch passed
          +1 javadoc 0m 55s the patch passed
          +1 unit 0m 53s hadoop-hdfs-client in the patch passed.
          +1 unit 64m 51s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          99m 4s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11538
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861182/HDFS-11538.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8d8cb74c00c4 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / b58777a
          Default Java 1.8.0_121
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18913/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt
          whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18913/artifact/patchprocess/whitespace-eol.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18913/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18913/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 28s Maven dependency ordering for branch +1 mvninstall 13m 7s trunk passed +1 compile 1m 22s trunk passed +1 checkstyle 0m 48s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 8s trunk passed +1 javadoc 1m 0s trunk passed 0 mvndep 0m 7s Maven dependency ordering for patch +1 mvninstall 1m 16s the patch passed +1 compile 1m 19s the patch passed +1 javac 1m 19s the patch passed -0 checkstyle 0m 46s hadoop-hdfs-project: The patch generated 43 new + 656 unchanged - 24 fixed = 699 total (was 680) +1 mvnsite 1m 19s the patch passed +1 mvneclipse 0m 20s the patch passed -1 whitespace 0m 0s The patch has 12 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply +1 findbugs 3m 20s the patch passed +1 javadoc 0m 55s the patch passed +1 unit 0m 53s hadoop-hdfs-client in the patch passed. +1 unit 64m 51s hadoop-hdfs in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 99m 4s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11538 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12861182/HDFS-11538.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8d8cb74c00c4 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / b58777a Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/18913/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project.txt whitespace https://builds.apache.org/job/PreCommit-HDFS-Build/18913/artifact/patchprocess/whitespace-eol.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/18913/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/18913/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          busbey Sean Busbey added a comment -

          Please do the test moves now as well. If we wait we're likely to have a substantial amount of time where no tests run against changes (based on the prior hadoop-hdfs-client changes)

          Show
          busbey Sean Busbey added a comment - Please do the test moves now as well. If we wait we're likely to have a substantial amount of time where no tests run against changes (based on the prior hadoop-hdfs-client changes)
          Hide
          HuafengWang Huafeng Wang added a comment -

          Hi Andrew Wang , I have uploaded my patch and please help to review it. I leave out the splits of tests in this patch and I think these can be done in following refactor.

          Show
          HuafengWang Huafeng Wang added a comment - Hi Andrew Wang , I have uploaded my patch and please help to review it. I leave out the splits of tests in this patch and I think these can be done in following refactor.
          Hide
          drankye Kai Zheng added a comment -

          Per offline discussion Huafeng Wang would work on this, so reassigned.

          Show
          drankye Kai Zheng added a comment - Per offline discussion Huafeng Wang would work on this, so reassigned.
          Hide
          drankye Kai Zheng added a comment -

          In the same org.apache.hadoop.hdfs.server.namenode.ha package, there also exist the following proxies that seem useful and maybe necessary to move together too. Guess this is what Allen meant.

          IPFailoverProxyProvider
          RequestHedgingProxyProvider
          
          Show
          drankye Kai Zheng added a comment - In the same org.apache.hadoop.hdfs.server.namenode.ha package, there also exist the following proxies that seem useful and maybe necessary to move together too. Guess this is what Allen meant. IPFailoverProxyProvider RequestHedgingProxyProvider
          Hide
          drankye Kai Zheng added a comment -

          More inputs from Andrew Wang in HDFS-11431:

          Hi Kai,

          This JIRA (HDFS-11431) adds hadoop-hdfs as a dependency of hadoop-hdfs-client (and thus also hadoop-client). This pulls all the server JARs back in. It was committed as a quick fix to get 2.8.0 released.

          As you noted, HDFS-11538 is intended as a better long term solution that moves CFPP to hadoop-hdfs-client, so it no longer needs to pull in the full hadoop-hdfs dependency (the server-side jar).

          This also highlights a significant lack of testing of the hdfs client artifact. The hdfs client split didn't also split the tests, so we have essentially no test coverage for the client module by itself. It'd be great for HDFS-11538 to include hdfs client unit tests to help address this.

          Show
          drankye Kai Zheng added a comment - More inputs from Andrew Wang in HDFS-11431 : Hi Kai, This JIRA ( HDFS-11431 ) adds hadoop-hdfs as a dependency of hadoop-hdfs-client (and thus also hadoop-client). This pulls all the server JARs back in. It was committed as a quick fix to get 2.8.0 released. As you noted, HDFS-11538 is intended as a better long term solution that moves CFPP to hadoop-hdfs-client, so it no longer needs to pull in the full hadoop-hdfs dependency (the server-side jar). This also highlights a significant lack of testing of the hdfs client artifact. The hdfs client split didn't also split the tests, so we have essentially no test coverage for the client module by itself. It'd be great for HDFS-11538 to include hdfs client unit tests to help address this.
          Hide
          drankye Kai Zheng added a comment -

          Hi Allen,

          I think there's another proxy provider in 3.x that should get moved over too.

          Sounds a good catch! What's that?

          Show
          drankye Kai Zheng added a comment - Hi Allen, I think there's another proxy provider in 3.x that should get moved over too. Sounds a good catch! What's that?
          Hide
          aw Allen Wittenauer added a comment -

          I think there's another proxy provider in 3.x that should get moved over too.

          Show
          aw Allen Wittenauer added a comment - I think there's another proxy provider in 3.x that should get moved over too.
          Hide
          andrew.wang Andrew Wang added a comment -

          WFM thanks Junping. Multi-release tracking like this is always difficult.

          Show
          andrew.wang Andrew Wang added a comment - WFM thanks Junping. Multi-release tracking like this is always difficult.
          Hide
          djp Junping Du added a comment -

          Add back 2.9.0 given discussion in HDFS-11431. However, it is not a blocker for 2.9.0 as workaround of HDFS-11431 still works for branch-2. If we have patch here before 2.9.0 going out, then we should revert HDFS-11431.

          Show
          djp Junping Du added a comment - Add back 2.9.0 given discussion in HDFS-11431 . However, it is not a blocker for 2.9.0 as workaround of HDFS-11431 still works for branch-2. If we have patch here before 2.9.0 going out, then we should revert HDFS-11431 .
          Hide
          djp Junping Du added a comment -

          Drop 2.9 as HDFS-11431 works well for branch-2.

          Show
          djp Junping Du added a comment - Drop 2.9 as HDFS-11431 works well for branch-2.

            People

            • Assignee:
              HuafengWang Huafeng Wang
              Reporter:
              andrew.wang Andrew Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development