Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9831

Document webhdfs retry configuration keys introduced by HDFS-5219/HDFS-5122

    Details

    • Hadoop Flags:
      Reviewed
    • Tags:
      webhdfs

      Description

      This ticket is opened to document the configuration keys introduced by HDFS-5219/HDFS-5122 for WebHdfs Retry. Both hdfs-default.xml and webhdfs.md should be updated with the usage of these keys.

      / WebHDFS retry policy
        public static final String  DFS_HTTP_CLIENT_RETRY_POLICY_ENABLED_KEY = "dfs.http.client.retry.policy.enabled";
        public static final boolean DFS_HTTP_CLIENT_RETRY_POLICY_ENABLED_DEFAULT = false;
        public static final String  DFS_HTTP_CLIENT_RETRY_POLICY_SPEC_KEY = "dfs.http.client.retry.policy.spec";
        public static final String  DFS_HTTP_CLIENT_RETRY_POLICY_SPEC_DEFAULT = "10000,6,60000,10"; //t1,n1,t2,n2,...
      public static final String  DFS_HTTP_CLIENT_FAILOVER_MAX_ATTEMPTS_KEY = "dfs.http.client.failover.max.attempts";
        public static final int    DFS_HTTP_CLIENT_FAILOVER_MAX_ATTEMPTS_DEFAULT = 15;
        public static final String  DFS_HTTP_CLIENT_RETRY_MAX_ATTEMPTS_KEY = "dfs.http.client.retry.max.attempts";
        public static final int    DFS_HTTP_CLIENT_RETRY_MAX_ATTEMPTS_DEFAULT = 10;
        public static final String  DFS_HTTP_CLIENT_FAILOVER_SLEEPTIME_BASE_KEY = "dfs.http.client.failover.sleep.base.millis";
        public static final int    DFS_HTTP_CLIENT_FAILOVER_SLEEPTIME_BASE_DEFAULT = 500;
        public static final String  DFS_HTTP_CLIENT_FAILOVER_SLEEPTIME_MAX_KEY = "dfs.http.client.failover.sleep.max.millis";
        public static final int    DFS_HTTP_CLIENT_FAILOVER_SLEEPTIME_MAX_DEFAULT = 15000;
      
      1. HDFS-9831.003.patch
        6 kB
        Xiaobing Zhou
      2. HDFS-9831.002.patch
        5 kB
        Xiaobing Zhou
      3. HDFS-9831.001.patch
        5 kB
        Xiaobing Zhou
      4. HDFS-9831.000.patch
        5 kB
        Xiaobing Zhou

        Activity

        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9378 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9378/)
        HDFS-9831. Document webhdfs retry configuration keys introduced by (xyao: rev eab52dfb35d1e876b69cf127ccf6cc07523ddf0b)

        • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9378 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9378/ ) HDFS-9831 . Document webhdfs retry configuration keys introduced by (xyao: rev eab52dfb35d1e876b69cf127ccf6cc07523ddf0b) hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Xiaobing Zhou for the contribution. I've committed the patch to trunk, branch-2, and branch-2.8.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Xiaobing Zhou for the contribution. I've committed the patch to trunk, branch-2, and branch-2.8.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 26s trunk passed
        +1 compile 0m 55s trunk passed with JDK v1.8.0_72
        +1 compile 0m 48s trunk passed with JDK v1.7.0_95
        +1 mvnsite 0m 59s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 javadoc 1m 23s trunk passed with JDK v1.8.0_72
        +1 javadoc 2m 5s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 53s the patch passed
        +1 compile 0m 53s the patch passed with JDK v1.8.0_72
        +1 javac 0m 53s the patch passed
        +1 compile 0m 47s the patch passed with JDK v1.7.0_95
        +1 javac 0m 47s the patch passed
        +1 mvnsite 0m 56s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 javadoc 1m 18s the patch passed with JDK v1.8.0_72
        +1 javadoc 2m 5s the patch passed with JDK v1.7.0_95
        -1 unit 94m 52s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
        -1 unit 80m 15s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        199m 1s



        Reason Tests
        JDK v1.8.0_72 Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts
          hadoop.hdfs.security.TestDelegationTokenForProxyUser
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.datanode.TestBlockReplacement
          hadoop.hdfs.server.datanode.TestDirectoryScanner
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
          hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
          hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness
          hadoop.hdfs.server.datanode.TestBlockReplacement



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790187/HDFS-9831.003.patch
        JIRA Issue HDFS-9831
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 589a77be120e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1aa0697
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14632/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14632/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14632/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14632/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14632/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14632/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 26s trunk passed +1 compile 0m 55s trunk passed with JDK v1.8.0_72 +1 compile 0m 48s trunk passed with JDK v1.7.0_95 +1 mvnsite 0m 59s trunk passed +1 mvneclipse 0m 15s trunk passed +1 javadoc 1m 23s trunk passed with JDK v1.8.0_72 +1 javadoc 2m 5s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 53s the patch passed +1 compile 0m 53s the patch passed with JDK v1.8.0_72 +1 javac 0m 53s the patch passed +1 compile 0m 47s the patch passed with JDK v1.7.0_95 +1 javac 0m 47s the patch passed +1 mvnsite 0m 56s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 1m 18s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 5s the patch passed with JDK v1.7.0_95 -1 unit 94m 52s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 80m 15s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 199m 1s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.web.TestWebHdfsTimeouts   hadoop.hdfs.security.TestDelegationTokenForProxyUser   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestBlockReplacement   hadoop.hdfs.server.datanode.TestDirectoryScanner JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.datanode.fsdataset.impl.TestFsDatasetImpl   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness   hadoop.hdfs.server.datanode.TestBlockReplacement Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790187/HDFS-9831.003.patch JIRA Issue HDFS-9831 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 589a77be120e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1aa0697 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14632/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14632/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14632/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14632/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14632/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14632/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        003 added anchor. Thanks Xiaoyu Yao.

        Show
        xiaobingo Xiaobing Zhou added a comment - 003 added anchor. Thanks Xiaoyu Yao .
        Hide
        xyao Xiaoyu Yao added a comment -

        Xiaobing Zhou, thanks for the update. We need to add an anchor for the new section in the top level directory.
        +1 after that being added.

        ...
            * [Cross-Site Request Forgery Prevention](#Cross-Site_Request_Forgery_Prevention)
            * [WebHDFS Retry Policy](#WebHDFS_Retry_Policy)
        
        Show
        xyao Xiaoyu Yao added a comment - Xiaobing Zhou , thanks for the update. We need to add an anchor for the new section in the top level directory. +1 after that being added. ... * [Cross-Site Request Forgery Prevention](#Cross-Site_Request_Forgery_Prevention) * [WebHDFS Retry Policy](#WebHDFS_Retry_Policy)
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 36s trunk passed
        +1 compile 0m 39s trunk passed with JDK v1.8.0_72
        +1 compile 0m 40s trunk passed with JDK v1.7.0_95
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 javadoc 1m 3s trunk passed with JDK v1.8.0_72
        +1 javadoc 1m 41s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 45s the patch passed
        +1 compile 0m 40s the patch passed with JDK v1.8.0_72
        +1 javac 0m 40s the patch passed
        +1 compile 0m 39s the patch passed with JDK v1.7.0_95
        +1 javac 0m 39s the patch passed
        +1 mvnsite 0m 47s the patch passed
        +1 mvneclipse 0m 10s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 javadoc 0m 59s the patch passed with JDK v1.8.0_72
        +1 javadoc 1m 44s the patch passed with JDK v1.7.0_95
        +1 unit 52m 55s hadoop-hdfs in the patch passed with JDK v1.8.0_72.
        -1 unit 52m 49s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 21s Patch does not generate ASF License warnings.
        125m 33s



        Reason Tests
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManagerSafeMode
          hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789984/HDFS-9831.002.patch
        JIRA Issue HDFS-9831
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 39ad7515cdf0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8808779
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14616/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14616/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14616/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14616/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 36s trunk passed +1 compile 0m 39s trunk passed with JDK v1.8.0_72 +1 compile 0m 40s trunk passed with JDK v1.7.0_95 +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 13s trunk passed +1 javadoc 1m 3s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 41s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 45s the patch passed +1 compile 0m 40s the patch passed with JDK v1.8.0_72 +1 javac 0m 40s the patch passed +1 compile 0m 39s the patch passed with JDK v1.7.0_95 +1 javac 0m 39s the patch passed +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 59s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 44s the patch passed with JDK v1.7.0_95 +1 unit 52m 55s hadoop-hdfs in the patch passed with JDK v1.8.0_72. -1 unit 52m 49s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 125m 33s Reason Tests JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.blockmanagement.TestBlockManagerSafeMode   hadoop.hdfs.server.blockmanagement.TestReconstructStripedBlocksWithRackAwareness Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789984/HDFS-9831.002.patch JIRA Issue HDFS-9831 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 39ad7515cdf0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8808779 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14616/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14616/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14616/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14616/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        I got confused with the structure of WebHDFS.md, 002 should fix it. Thanks Xiaoyu Yao.

        Show
        xiaobingo Xiaobing Zhou added a comment - I got confused with the structure of WebHDFS.md, 002 should fix it. Thanks Xiaoyu Yao .
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Xiaobing Zhou for the update. One more issue found on the rendered webhdfs.html (changes in webhdfs.md)

        I don't think we should put "The following properties control WebHDFS retry and failover policy." and the retry keys under the "Cross-Site Request Forgery Prevention" section. Can you add this as a separate section like below?

        WebHDFS Retry Policy
        -------------------------------------
        
        WebHDFS supports an optional, configurable retry policy for resilient copy of large files that could timeout, or copy file between HA clusters that could failover during the copy.
        
        The following properties control WebHDFS retry and failover policy.
        ...
        
        
        Show
        xyao Xiaoyu Yao added a comment - Thanks Xiaobing Zhou for the update. One more issue found on the rendered webhdfs.html (changes in webhdfs.md) I don't think we should put "The following properties control WebHDFS retry and failover policy." and the retry keys under the "Cross-Site Request Forgery Prevention" section. Can you add this as a separate section like below? WebHDFS Retry Policy ------------------------------------- WebHDFS supports an optional, configurable retry policy for resilient copy of large files that could timeout, or copy file between HA clusters that could failover during the copy. The following properties control WebHDFS retry and failover policy. ...
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 25s trunk passed
        +1 compile 0m 47s trunk passed with JDK v1.8.0_72
        +1 compile 0m 43s trunk passed with JDK v1.7.0_95
        +1 mvnsite 0m 54s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 javadoc 1m 8s trunk passed with JDK v1.8.0_72
        +1 javadoc 1m 50s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 44s the patch passed
        +1 compile 0m 40s the patch passed with JDK v1.8.0_72
        +1 javac 0m 40s the patch passed
        +1 compile 0m 38s the patch passed with JDK v1.7.0_95
        +1 javac 0m 38s the patch passed
        +1 mvnsite 0m 50s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        +1 javadoc 1m 8s the patch passed with JDK v1.8.0_72
        +1 javadoc 1m 43s the patch passed with JDK v1.7.0_95
        -1 unit 53m 26s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
        -1 unit 50m 50s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 21s Patch does not generate ASF License warnings.
        125m 39s



        Reason Tests
        JDK v1.8.0_72 Failed junit tests hadoop.hdfs.TestFileCreationDelete
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789834/HDFS-9831.001.patch
        JIRA Issue HDFS-9831
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 4d2e4c0ef57b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / dbbfc58
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14608/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14608/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14608/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14608/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14608/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14608/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 25s trunk passed +1 compile 0m 47s trunk passed with JDK v1.8.0_72 +1 compile 0m 43s trunk passed with JDK v1.7.0_95 +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 14s trunk passed +1 javadoc 1m 8s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 50s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 44s the patch passed +1 compile 0m 40s the patch passed with JDK v1.8.0_72 +1 javac 0m 40s the patch passed +1 compile 0m 38s the patch passed with JDK v1.7.0_95 +1 javac 0m 38s the patch passed +1 mvnsite 0m 50s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 javadoc 1m 8s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 43s the patch passed with JDK v1.7.0_95 -1 unit 53m 26s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 50m 50s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 125m 39s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.TestFileCreationDelete JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789834/HDFS-9831.001.patch JIRA Issue HDFS-9831 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 4d2e4c0ef57b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / dbbfc58 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14608/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14608/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14608/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14608/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14608/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14608/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        001 added the use cases, thanks Xiaoyu Yao for review.

        Show
        xiaobingo Xiaobing Zhou added a comment - 001 added the use cases, thanks Xiaoyu Yao for review.
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Xiaobing Zhou for working on this. The patch looks good to me. One suggestion: can you add some description of the use cases that need to enable the WebHDFS retry policy in hdfs-site.xml. For example,

        If "true", enable the retry policy of WebHDFS client. This can be useful when using WebHDFS to

        • copy large files between clusters that could timeout or
        • copy files between HA clusters that could failover during the copy.
        2834	<property>
        2835	  <name>dfs.http.client.retry.policy.enabled</name>
        2836	  <value>false</value>
        2837	  <description>
        2838	    If "true", enable the retry policy of WebHDFS client.
        2839	    If "false", retry policy is turned off.
        2840	  </description>
        2841	</property>
        
        Show
        xyao Xiaoyu Yao added a comment - Thanks Xiaobing Zhou for working on this. The patch looks good to me. One suggestion: can you add some description of the use cases that need to enable the WebHDFS retry policy in hdfs-site.xml. For example, If "true", enable the retry policy of WebHDFS client. This can be useful when using WebHDFS to copy large files between clusters that could timeout or copy files between HA clusters that could failover during the copy. 2834 <property> 2835 <name>dfs.http.client.retry.policy.enabled</name> 2836 <value> false </value> 2837 <description> 2838 If " true " , enable the retry policy of WebHDFS client. 2839 If " false " , retry policy is turned off. 2840 </description> 2841 </property>
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 22s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 11s trunk passed
        +1 compile 0m 52s trunk passed with JDK v1.8.0_72
        +1 compile 0m 44s trunk passed with JDK v1.7.0_95
        +1 mvnsite 0m 54s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 javadoc 1m 12s trunk passed with JDK v1.8.0_72
        +1 javadoc 1m 53s trunk passed with JDK v1.7.0_95
        +1 mvninstall 0m 49s the patch passed
        +1 compile 0m 50s the patch passed with JDK v1.8.0_72
        +1 javac 0m 50s the patch passed
        +1 compile 0m 42s the patch passed with JDK v1.7.0_95
        +1 javac 0m 42s the patch passed
        +1 mvnsite 0m 52s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        +1 javadoc 1m 15s the patch passed with JDK v1.8.0_72
        +1 javadoc 1m 49s the patch passed with JDK v1.7.0_95
        -1 unit 66m 46s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
        -1 unit 60m 5s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 20s Patch does not generate ASF License warnings.
        148m 56s



        Reason Tests
        JDK v1.8.0_72 Failed junit tests hadoop.hdfs.TestFileAppend
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestRollingUpgrade
          hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789106/HDFS-9831.000.patch
        JIRA Issue HDFS-9831
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 8aa72fd32871 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 66289a3
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14576/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14576/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14576/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14576/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14576/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14576/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 11s trunk passed +1 compile 0m 52s trunk passed with JDK v1.8.0_72 +1 compile 0m 44s trunk passed with JDK v1.7.0_95 +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 13s trunk passed +1 javadoc 1m 12s trunk passed with JDK v1.8.0_72 +1 javadoc 1m 53s trunk passed with JDK v1.7.0_95 +1 mvninstall 0m 49s the patch passed +1 compile 0m 50s the patch passed with JDK v1.8.0_72 +1 javac 0m 50s the patch passed +1 compile 0m 42s the patch passed with JDK v1.7.0_95 +1 javac 0m 42s the patch passed +1 mvnsite 0m 52s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 javadoc 1m 15s the patch passed with JDK v1.8.0_72 +1 javadoc 1m 49s the patch passed with JDK v1.7.0_95 -1 unit 66m 46s hadoop-hdfs in the patch failed with JDK v1.8.0_72. -1 unit 60m 5s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 20s Patch does not generate ASF License warnings. 148m 56s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.TestFileAppend JDK v1.7.0_95 Failed junit tests hadoop.hdfs.TestRollingUpgrade   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789106/HDFS-9831.000.patch JIRA Issue HDFS-9831 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 8aa72fd32871 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 66289a3 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14576/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/14576/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14576/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HDFS-Build/14576/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14576/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14576/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xiaobingo Xiaobing Zhou added a comment -

        I posted 000 patch, kindly review please, thanks.

        Show
        xiaobingo Xiaobing Zhou added a comment - I posted 000 patch, kindly review please, thanks.

          People

          • Assignee:
            xiaobingo Xiaobing Zhou
            Reporter:
            xyao Xiaoyu Yao
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development