Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9887

WebHdfs socket timeouts should be configurable

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Added new configuration options: dfs.webhdfs.socket.connect-timeout and dfs.webhdfs.socket.read-timeout both defaulting to 60s.
    • Flags:
      Patch

      Description

      WebHdfs client connections use sockets with fixed timeouts of 60 seconds to connect, and 60 seconds for reads.
      This is a problem because I am trying to use WebHdfs to access an archive storage system which can take minutes to hours to return the requested data over WebHdfs.

      The fix is to add new configuration file options to allow these 60s defaults to be customised in hdfs-site.xml.

      If the new configuration options are not present, the behavior is unchanged from before.

      1. HADOOP-12827.001.patch
        6 kB
        Austin Donnelly
      2. HADOOP-12827.002.patch
        11 kB
        Austin Donnelly
      3. HADOOP-12827.002.patch
        11 kB
        Austin Donnelly
      4. HADOOP-12827.002.patch
        11 kB
        Austin Donnelly
      5. HADOOP-12827.003.patch
        11 kB
        Chris Douglas
      6. HADOOP-12827.004.patch
        11 kB
        Chris Douglas

        Activity

        Hide
        and1000 Austin Donnelly added a comment -

        Here's a patch which fixes this issue.

        Testing: no new test code added, since this only adds two config options.

        Manual testing: I tested the following scenarios:

        • New config not present. Client timeout verified unchanged at 60s.
        • New config present: 30s for connect timeout, 2m for read timeout:
        • WebHdfs server not listening => client timeout at 30s as expected.
        • WebHdfs server up, but modified to stall data => client timeout at 2m as expected.
        • WebHdfs server up, operating normally => client operates normally.
        • Also tested with distCp. Before patch, some transfers would timeout, after patch, set longer (30m) read timeout, and distCp completes without timeouts.
        Show
        and1000 Austin Donnelly added a comment - Here's a patch which fixes this issue. Testing: no new test code added, since this only adds two config options. Manual testing: I tested the following scenarios: New config not present. Client timeout verified unchanged at 60s. New config present: 30s for connect timeout, 2m for read timeout: WebHdfs server not listening => client timeout at 30s as expected. WebHdfs server up, but modified to stall data => client timeout at 2m as expected. WebHdfs server up, operating normally => client operates normally. Also tested with distCp. Before patch, some transfers would timeout, after patch, set longer (30m) read timeout, and distCp completes without timeouts.
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Austin Donnelly for reporting the issue and propose the fix. We have seen similar customer issues recently and think about approach similar to the one you proposed here. We proceed with an existing webhdfs retry mechanism introduced by HDFS-5219/HDFS-5122 to work around it without code changes, which is orthogonal to approach proposed here.

        The patch looks good to me overall. I like the usage of time unit with suffix instead of hard-code time duration to ms or seconds. Below are some of my suggestions:

        1. We need to document the new keys in hdfs-default.xml?
        2. Your manual test result looks good to me. Can you add them to unit tests by enhancing TestWebHdfsTimeouts with the new configuration keys?

        Show
        xyao Xiaoyu Yao added a comment - Thanks Austin Donnelly for reporting the issue and propose the fix. We have seen similar customer issues recently and think about approach similar to the one you proposed here. We proceed with an existing webhdfs retry mechanism introduced by HDFS-5219 / HDFS-5122 to work around it without code changes, which is orthogonal to approach proposed here. The patch looks good to me overall. I like the usage of time unit with suffix instead of hard-code time duration to ms or seconds. Below are some of my suggestions: 1. We need to document the new keys in hdfs-default.xml? 2. Your manual test result looks good to me. Can you add them to unit tests by enhancing TestWebHdfsTimeouts with the new configuration keys?
        Hide
        xyao Xiaoyu Yao added a comment -

        One disadvantage of configurable timeout key is it is hard to tune. But I agree it is complementary to retry when the default timeout does not work.

        Show
        xyao Xiaoyu Yao added a comment - One disadvantage of configurable timeout key is it is hard to tune. But I agree it is complementary to retry when the default timeout does not work.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        0 mvndep 0m 11s Maven dependency ordering for branch
        +1 mvninstall 8m 17s trunk passed
        +1 compile 1m 35s trunk passed with JDK v1.8.0_72
        +1 compile 1m 33s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 30s trunk passed
        +1 mvnsite 1m 41s trunk passed
        +1 mvneclipse 0m 29s trunk passed
        +1 findbugs 4m 27s trunk passed
        +1 javadoc 1m 39s trunk passed with JDK v1.8.0_72
        +1 javadoc 2m 38s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 29s the patch passed
        +1 compile 1m 42s the patch passed with JDK v1.8.0_72
        +1 javac 1m 42s the patch passed
        +1 compile 1m 38s the patch passed with JDK v1.7.0_95
        +1 javac 1m 38s the patch passed
        +1 checkstyle 0m 29s the patch passed
        +1 mvnsite 1m 39s the patch passed
        +1 mvneclipse 0m 24s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 4m 42s the patch passed
        +1 javadoc 1m 45s the patch passed with JDK v1.8.0_72
        +1 javadoc 2m 28s the patch passed with JDK v1.7.0_95
        +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72.
        -1 unit 64m 39s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
        +1 unit 1m 2s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
        +1 unit 55m 54s hadoop-hdfs in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 24s Patch does not generate ASF License warnings.
        165m 20s



        Reason Tests
        JDK v1.8.0_72 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789010/HADOOP-12827.001.patch
        JIRA Issue HADOOP-12827
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 39f90f24a858 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 5e7d4d5
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8684/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8684/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8684/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8684/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 8m 17s trunk passed +1 compile 1m 35s trunk passed with JDK v1.8.0_72 +1 compile 1m 33s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 30s trunk passed +1 mvnsite 1m 41s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 4m 27s trunk passed +1 javadoc 1m 39s trunk passed with JDK v1.8.0_72 +1 javadoc 2m 38s trunk passed with JDK v1.7.0_95 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 29s the patch passed +1 compile 1m 42s the patch passed with JDK v1.8.0_72 +1 javac 1m 42s the patch passed +1 compile 1m 38s the patch passed with JDK v1.7.0_95 +1 javac 1m 38s the patch passed +1 checkstyle 0m 29s the patch passed +1 mvnsite 1m 39s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 4m 42s the patch passed +1 javadoc 1m 45s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 28s the patch passed with JDK v1.7.0_95 +1 unit 1m 1s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72. -1 unit 64m 39s hadoop-hdfs in the patch failed with JDK v1.8.0_72. +1 unit 1m 2s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. +1 unit 55m 54s hadoop-hdfs in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 165m 20s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789010/HADOOP-12827.001.patch JIRA Issue HADOOP-12827 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 39f90f24a858 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5e7d4d5 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8684/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8684/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8684/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8684/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        and1000 Austin Donnelly added a comment -

        New patch to address issues:
        1) added documentation to hdfs-default.xml
        2) added new tests covering these config keys

        Show
        and1000 Austin Donnelly added a comment - New patch to address issues: 1) added documentation to hdfs-default.xml 2) added new tests covering these config keys
        Hide
        and1000 Austin Donnelly added a comment -

        [newbie] How do I get Hadoop QA bot to evaluate my latest patch? I've tried changing the Status earlier today, but nothing seems to have happened. Any suggestions?

        Show
        and1000 Austin Donnelly added a comment - [newbie] How do I get Hadoop QA bot to evaluate my latest patch? I've tried changing the Status earlier today, but nothing seems to have happened. Any suggestions?
        Hide
        xyao Xiaoyu Yao added a comment -

        Austin Donnelly, "Cancel Patch" and Resubmit will do the trick. You can check the precommit build queue here: https://builds.apache.org/job/PreCommit-HADOOP-Build/.

        Show
        xyao Xiaoyu Yao added a comment - Austin Donnelly , "Cancel Patch" and Resubmit will do the trick. You can check the precommit build queue here: https://builds.apache.org/job/PreCommit-HADOOP-Build/ .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 10s Maven dependency ordering for branch
        +1 mvninstall 6m 38s trunk passed
        +1 compile 1m 16s trunk passed with JDK v1.8.0_72
        +1 compile 1m 20s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 26s trunk passed
        +1 mvnsite 1m 24s trunk passed
        +1 mvneclipse 0m 25s trunk passed
        +1 findbugs 3m 36s trunk passed
        +1 javadoc 1m 25s trunk passed with JDK v1.8.0_72
        +1 javadoc 2m 7s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 1m 16s the patch passed
        +1 compile 1m 13s the patch passed with JDK v1.8.0_72
        +1 javac 1m 13s the patch passed
        +1 compile 1m 17s the patch passed with JDK v1.7.0_95
        +1 javac 1m 17s the patch passed
        +1 checkstyle 0m 24s the patch passed
        +1 mvnsite 1m 22s the patch passed
        +1 mvneclipse 0m 22s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 57s the patch passed
        +1 javadoc 1m 21s the patch passed with JDK v1.8.0_72
        +1 javadoc 2m 4s the patch passed with JDK v1.7.0_95
        +1 unit 0m 50s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72.
        -1 unit 53m 53s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
        +1 unit 0m 56s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
        -1 unit 52m 1s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 21s Patch does not generate ASF License warnings.
        142m 40s



        Reason Tests
        JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade
        JDK v1.7.0_95 Failed junit tests hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs
          hadoop.hdfs.TestDFSClientExcludedNodes
          hadoop.hdfs.TestCrcCorruption



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789636/HADOOP-12827.002.patch
        JIRA Issue HADOOP-12827
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 8df96a786720 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 2e76c2f
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8709/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8709/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8709/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8709/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8709/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8709/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 6m 38s trunk passed +1 compile 1m 16s trunk passed with JDK v1.8.0_72 +1 compile 1m 20s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 26s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 36s trunk passed +1 javadoc 1m 25s trunk passed with JDK v1.8.0_72 +1 javadoc 2m 7s trunk passed with JDK v1.7.0_95 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 16s the patch passed +1 compile 1m 13s the patch passed with JDK v1.8.0_72 +1 javac 1m 13s the patch passed +1 compile 1m 17s the patch passed with JDK v1.7.0_95 +1 javac 1m 17s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 1m 22s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 57s the patch passed +1 javadoc 1m 21s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 4s the patch passed with JDK v1.7.0_95 +1 unit 0m 50s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72. -1 unit 53m 53s hadoop-hdfs in the patch failed with JDK v1.8.0_72. +1 unit 0m 56s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 52m 1s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 142m 40s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade JDK v1.7.0_95 Failed junit tests hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs   hadoop.hdfs.TestDFSClientExcludedNodes   hadoop.hdfs.TestCrcCorruption Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12789636/HADOOP-12827.002.patch JIRA Issue HADOOP-12827 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 8df96a786720 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2e76c2f Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8709/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8709/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8709/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8709/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8709/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8709/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        and1000 Austin Donnelly added a comment -

        The test failures are unexpected, as they are not in tests which I changed. Is this perhaps due to noisy tests?
        The failures were due to Bind() failing due to port-in-use problem: is this due to failure to isolate tests which are expected to be independent?
        Resubmitting the same patch to see if the same tests fail deterministically.

        Show
        and1000 Austin Donnelly added a comment - The test failures are unexpected, as they are not in tests which I changed. Is this perhaps due to noisy tests? The failures were due to Bind() failing due to port-in-use problem: is this due to failure to isolate tests which are expected to be independent? Resubmitting the same patch to see if the same tests fail deterministically.
        Hide
        xyao Xiaoyu Yao added a comment -

        Austin Donnelly, thanks for the update. A few additional comments:

        1. In hdfs-dfault.xml, can rephrase the following into something like "Value is recommended to followed by a unit specifier.... If no unit specified is given, the default will be milliseconds."

        Value is followed by a unit specifier: ns, us, ms, s, m,
        2328	    h, d for nanoseconds, microseconds, milliseconds, seconds,
        2329	    minutes, hours, days respectively.
        

        2. In WebHDFS.md, I don't think we should put the dfs.webhdfs.socket.connect-timeout and dfs.webhdfs.socket.read-timeout in the oauth2 table. Can you add a new section with its own anchor for them? Here is an example: https://issues.apache.org/jira/secure/attachment/12790187/HDFS-9831.003.patch#file-1

        3. Please update the patch number when you resubmit a new patch and also remove the duplicate 002 patches to avoid confusion.

        Show
        xyao Xiaoyu Yao added a comment - Austin Donnelly , thanks for the update. A few additional comments: 1. In hdfs-dfault.xml, can rephrase the following into something like "Value is recommended to followed by a unit specifier.... If no unit specified is given, the default will be milliseconds." Value is followed by a unit specifier: ns, us, ms, s, m, 2328 h, d for nanoseconds, microseconds, milliseconds, seconds, 2329 minutes, hours, days respectively. 2. In WebHDFS.md, I don't think we should put the dfs.webhdfs.socket.connect-timeout and dfs.webhdfs.socket.read-timeout in the oauth2 table. Can you add a new section with its own anchor for them? Here is an example: https://issues.apache.org/jira/secure/attachment/12790187/HDFS-9831.003.patch#file-1 3. Please update the patch number when you resubmit a new patch and also remove the duplicate 002 patches to avoid confusion.
        Hide
        xyao Xiaoyu Yao added a comment -

        There is a typo in my suggested revision of hdfs-default.xml, the correct one should be:

        1. In hdfs-dfault.xml, can rephrase the following into something like "Value is recommended to followed by a unit specifier.... If no unit specifier is given, the default will be milliseconds."

        Show
        xyao Xiaoyu Yao added a comment - There is a typo in my suggested revision of hdfs-default.xml, the correct one should be: 1. In hdfs-dfault.xml, can rephrase the following into something like "Value is recommended to followed by a unit specifier.... If no unit specifier is given, the default will be milliseconds."
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 10s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 26s Maven dependency ordering for branch
        +1 mvninstall 7m 23s trunk passed
        +1 compile 1m 23s trunk passed with JDK v1.8.0_72
        +1 compile 1m 26s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 28s trunk passed
        +1 mvnsite 1m 32s trunk passed
        +1 mvneclipse 0m 26s trunk passed
        +1 findbugs 3m 46s trunk passed
        +1 javadoc 1m 32s trunk passed with JDK v1.8.0_72
        +1 javadoc 2m 17s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 1m 21s the patch passed
        +1 compile 1m 19s the patch passed with JDK v1.8.0_72
        +1 javac 1m 19s the patch passed
        +1 compile 1m 23s the patch passed with JDK v1.7.0_95
        +1 javac 1m 23s the patch passed
        +1 checkstyle 0m 25s the patch passed
        +1 mvnsite 1m 27s the patch passed
        +1 mvneclipse 0m 24s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 4m 21s the patch passed
        +1 javadoc 1m 26s the patch passed with JDK v1.8.0_72
        +1 javadoc 2m 20s the patch passed with JDK v1.7.0_95
        +1 unit 0m 53s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72.
        -1 unit 63m 19s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
        +1 unit 1m 4s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
        -1 unit 66m 25s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 43s Patch does not generate ASF License warnings.
        170m 22s



        Reason Tests
        JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation
          hadoop.hdfs.TestFileAppend
          hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits
        JDK v1.7.0_95 Failed junit tests hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790150/HADOOP-12827.002.patch
        JIRA Issue HADOOP-12827
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux d1fa9e21ef35 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d7fdec1
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8729/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8729/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8729/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8729/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8729/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8729/console
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 26s Maven dependency ordering for branch +1 mvninstall 7m 23s trunk passed +1 compile 1m 23s trunk passed with JDK v1.8.0_72 +1 compile 1m 26s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 32s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 46s trunk passed +1 javadoc 1m 32s trunk passed with JDK v1.8.0_72 +1 javadoc 2m 17s trunk passed with JDK v1.7.0_95 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 21s the patch passed +1 compile 1m 19s the patch passed with JDK v1.8.0_72 +1 javac 1m 19s the patch passed +1 compile 1m 23s the patch passed with JDK v1.7.0_95 +1 javac 1m 23s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 1m 27s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 21s the patch passed +1 javadoc 1m 26s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 20s the patch passed with JDK v1.7.0_95 +1 unit 0m 53s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72. -1 unit 63m 19s hadoop-hdfs in the patch failed with JDK v1.8.0_72. +1 unit 1m 4s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 66m 25s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 43s Patch does not generate ASF License warnings. 170m 22s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation   hadoop.hdfs.TestFileAppend   hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits JDK v1.7.0_95 Failed junit tests hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790150/HADOOP-12827.002.patch JIRA Issue HADOOP-12827 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux d1fa9e21ef35 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d7fdec1 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8729/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8729/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8729/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt https://builds.apache.org/job/PreCommit-HADOOP-Build/8729/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8729/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8729/console Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        chris.douglas Chris Douglas added a comment -

        In hdfs-dfault.xml, can rephrase the following into something like "Value is recommended to followed by a unit specifier.... If no unit specified is given, the default will be milliseconds."

        The user should always provide a unit, as Configuration::getTimeDuration complains if it is not provided. Would you mind if the default unit remained unspecified?

        In WebHDFS.md, I don't think we should put the dfs.webhdfs.socket.connect-timeout and dfs.webhdfs.socket.read-timeout in the oauth2 table. Can you add a new section with its own anchor for them?

        Updated patch, grouped the timeouts with other WebHDFS properties.

        The patch looks good to me. Xiaoyu Yao, anything else before commit?

        Show
        chris.douglas Chris Douglas added a comment - In hdfs-dfault.xml, can rephrase the following into something like "Value is recommended to followed by a unit specifier.... If no unit specified is given, the default will be milliseconds." The user should always provide a unit, as Configuration::getTimeDuration complains if it is not provided. Would you mind if the default unit remained unspecified? In WebHDFS.md, I don't think we should put the dfs.webhdfs.socket.connect-timeout and dfs.webhdfs.socket.read-timeout in the oauth2 table. Can you add a new section with its own anchor for them? Updated patch, grouped the timeouts with other WebHDFS properties. The patch looks good to me. Xiaoyu Yao , anything else before commit?
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 19s Maven dependency ordering for branch
        +1 mvninstall 6m 46s trunk passed
        +1 compile 1m 16s trunk passed with JDK v1.8.0_72
        +1 compile 1m 22s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 25s trunk passed
        +1 mvnsite 1m 27s trunk passed
        +1 mvneclipse 0m 27s trunk passed
        +1 findbugs 3m 30s trunk passed
        +1 javadoc 1m 31s trunk passed with JDK v1.8.0_72
        +1 javadoc 2m 14s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 1m 18s the patch passed
        +1 compile 1m 15s the patch passed with JDK v1.8.0_72
        +1 javac 1m 15s the patch passed
        +1 compile 1m 21s the patch passed with JDK v1.7.0_95
        +1 javac 1m 21s the patch passed
        +1 checkstyle 0m 25s the patch passed
        +1 mvnsite 1m 24s the patch passed
        +1 mvneclipse 0m 24s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        +1 findbugs 4m 0s the patch passed
        +1 javadoc 1m 22s the patch passed with JDK v1.8.0_72
        +1 javadoc 2m 10s the patch passed with JDK v1.7.0_95
        +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72.
        +1 unit 67m 54s hadoop-hdfs in the patch passed with JDK v1.8.0_72.
        +1 unit 0m 58s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
        +1 unit 71m 4s hadoop-hdfs in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 21s Patch does not generate ASF License warnings.
        176m 44s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790846/HADOOP-12827.003.patch
        JIRA Issue HADOOP-12827
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 84ac7d5679af 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f2ba7da
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8760/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8760/console
        Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 19s Maven dependency ordering for branch +1 mvninstall 6m 46s trunk passed +1 compile 1m 16s trunk passed with JDK v1.8.0_72 +1 compile 1m 22s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 25s trunk passed +1 mvnsite 1m 27s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 3m 30s trunk passed +1 javadoc 1m 31s trunk passed with JDK v1.8.0_72 +1 javadoc 2m 14s trunk passed with JDK v1.7.0_95 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 18s the patch passed +1 compile 1m 15s the patch passed with JDK v1.8.0_72 +1 javac 1m 15s the patch passed +1 compile 1m 21s the patch passed with JDK v1.7.0_95 +1 javac 1m 21s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 1m 24s the patch passed +1 mvneclipse 0m 24s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 4m 0s the patch passed +1 javadoc 1m 22s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 10s the patch passed with JDK v1.7.0_95 +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72. +1 unit 67m 54s hadoop-hdfs in the patch passed with JDK v1.8.0_72. +1 unit 0m 58s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. +1 unit 71m 4s hadoop-hdfs in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 176m 44s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790846/HADOOP-12827.003.patch JIRA Issue HADOOP-12827 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 84ac7d5679af 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f2ba7da Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8760/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8760/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Chris Douglas for updating the patch.

        In hdfs-dfault.xml, can rephrase the following into something like "Value is recommended to followed by a unit specifier.... If no unit specified is given, the default will be milliseconds."

        The user should always provide a unit, as Configuration::getTimeDuration complains if it is not provided. Would you mind if the default unit remained unspecified?

        I would suggest having the expected behavior documented when time unit unspecified. Otherwise, we should add a different version of Configuration::getTimeDuration() that does not take default unit parameter and throw exception instead of only logging a warn.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Chris Douglas for updating the patch. In hdfs-dfault.xml, can rephrase the following into something like "Value is recommended to followed by a unit specifier.... If no unit specified is given, the default will be milliseconds." The user should always provide a unit, as Configuration::getTimeDuration complains if it is not provided. Would you mind if the default unit remained unspecified? I would suggest having the expected behavior documented when time unit unspecified. Otherwise, we should add a different version of Configuration::getTimeDuration() that does not take default unit parameter and throw exception instead of only logging a warn.
        Hide
        chris.douglas Chris Douglas added a comment - - edited

        OK. Added a line to hdfs-default.xml explaining the default unit.

        Show
        chris.douglas Chris Douglas added a comment - - edited OK. Added a line to hdfs-default.xml explaining the default unit.
        Hide
        and1000 Austin Donnelly added a comment -

        Looks good to me. Thanks for the updates Chris Douglas.

        Show
        and1000 Austin Donnelly added a comment - Looks good to me. Thanks for the updates Chris Douglas .
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 31s Maven dependency ordering for branch
        +1 mvninstall 7m 0s trunk passed
        +1 compile 1m 18s trunk passed with JDK v1.8.0_72
        +1 compile 1m 19s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 27s trunk passed
        +1 mvnsite 1m 26s trunk passed
        +1 mvneclipse 0m 25s trunk passed
        +1 findbugs 3m 33s trunk passed
        +1 javadoc 1m 26s trunk passed with JDK v1.8.0_72
        +1 javadoc 2m 11s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 8s Maven dependency ordering for patch
        +1 mvninstall 1m 16s the patch passed
        +1 compile 1m 19s the patch passed with JDK v1.8.0_72
        +1 javac 1m 19s the patch passed
        +1 compile 1m 19s the patch passed with JDK v1.7.0_95
        +1 javac 1m 19s the patch passed
        +1 checkstyle 0m 25s the patch passed
        +1 mvnsite 1m 24s the patch passed
        +1 mvneclipse 0m 22s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 59s the patch passed
        +1 javadoc 1m 26s the patch passed with JDK v1.8.0_72
        +1 javadoc 2m 10s the patch passed with JDK v1.7.0_95
        +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72.
        -1 unit 55m 6s hadoop-hdfs in the patch failed with JDK v1.8.0_72.
        +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
        +1 unit 52m 7s hadoop-hdfs in the patch passed with JDK v1.7.0_95.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        145m 41s



        Reason Tests
        JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.balancer.TestBalancer
          hadoop.hdfs.TestFileAppend



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790889/HADOOP-12827.004.patch
        JIRA Issue HADOOP-12827
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 2d3cbef3095f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 27e0681
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8762/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
        unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8762/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8762/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8762/console
        Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 31s Maven dependency ordering for branch +1 mvninstall 7m 0s trunk passed +1 compile 1m 18s trunk passed with JDK v1.8.0_72 +1 compile 1m 19s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 27s trunk passed +1 mvnsite 1m 26s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 3m 33s trunk passed +1 javadoc 1m 26s trunk passed with JDK v1.8.0_72 +1 javadoc 2m 11s trunk passed with JDK v1.7.0_95 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 1m 16s the patch passed +1 compile 1m 19s the patch passed with JDK v1.8.0_72 +1 javac 1m 19s the patch passed +1 compile 1m 19s the patch passed with JDK v1.7.0_95 +1 javac 1m 19s the patch passed +1 checkstyle 0m 25s the patch passed +1 mvnsite 1m 24s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 59s the patch passed +1 javadoc 1m 26s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 10s the patch passed with JDK v1.7.0_95 +1 unit 1m 0s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72. -1 unit 55m 6s hadoop-hdfs in the patch failed with JDK v1.8.0_72. +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. +1 unit 52m 7s hadoop-hdfs in the patch passed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 145m 41s Reason Tests JDK v1.8.0_72 Failed junit tests hadoop.hdfs.server.balancer.TestBalancer   hadoop.hdfs.TestFileAppend Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790889/HADOOP-12827.004.patch JIRA Issue HADOOP-12827 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 2d3cbef3095f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 27e0681 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8762/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8762/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.8.0_72.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8762/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8762/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        xyao Xiaoyu Yao added a comment -

        +1 patch v004, I will commit it shortly.

        Show
        xyao Xiaoyu Yao added a comment - +1 patch v004, I will commit it shortly.
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Austin Donnelly and Chris Douglas for the contribution. I've commit the patch to trunk, branch-2 and branch2.8.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Austin Donnelly and Chris Douglas for the contribution. I've commit the patch to trunk, branch-2 and branch2.8.
        Hide
        and1000 Austin Donnelly added a comment -

        Thank you!

        Show
        and1000 Austin Donnelly added a comment - Thank you!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9411 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9411/)
        HDFS-9887. WebHdfs socket timeouts should be configurable. Contributed (xyao: rev 5abf051249d485313dfffc6aeff6f81c0da1f623)

        • hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
        • hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
        • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTimeouts.java
        • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/URLConnectionFactory.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9411 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9411/ ) HDFS-9887 . WebHdfs socket timeouts should be configurable. Contributed (xyao: rev 5abf051249d485313dfffc6aeff6f81c0da1f623) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTimeouts.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/URLConnectionFactory.java
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 25s Maven dependency ordering for branch
        +1 mvninstall 6m 48s trunk passed
        +1 compile 1m 14s trunk passed with JDK v1.8.0_72
        +1 compile 1m 19s trunk passed with JDK v1.7.0_95
        +1 checkstyle 0m 27s trunk passed
        +1 mvnsite 1m 27s trunk passed
        +1 mvneclipse 0m 26s trunk passed
        +1 findbugs 3m 33s trunk passed
        +1 javadoc 1m 24s trunk passed with JDK v1.8.0_72
        +1 javadoc 2m 15s trunk passed with JDK v1.7.0_95
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 1m 17s the patch passed
        +1 compile 1m 13s the patch passed with JDK v1.8.0_72
        +1 javac 1m 13s the patch passed
        +1 compile 1m 20s the patch passed with JDK v1.7.0_95
        +1 javac 1m 20s the patch passed
        +1 checkstyle 0m 24s the patch passed
        +1 mvnsite 1m 22s the patch passed
        +1 mvneclipse 0m 22s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 59s the patch passed
        +1 javadoc 1m 22s the patch passed with JDK v1.8.0_72
        +1 javadoc 2m 7s the patch passed with JDK v1.7.0_95
        +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72.
        +1 unit 67m 50s hadoop-hdfs in the patch passed with JDK v1.8.0_72.
        +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95.
        -1 unit 73m 33s hadoop-hdfs in the patch failed with JDK v1.7.0_95.
        +1 asflicense 0m 23s Patch does not generate ASF License warnings.
        178m 53s



        Reason Tests
        JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot
        JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790889/HADOOP-12827.004.patch
        JIRA Issue HDFS-9887
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux cf8e35fbf01f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 27e0681
        Default Java 1.7.0_95
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/14687/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14687/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt
        JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14687/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14687/console
        Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 25s Maven dependency ordering for branch +1 mvninstall 6m 48s trunk passed +1 compile 1m 14s trunk passed with JDK v1.8.0_72 +1 compile 1m 19s trunk passed with JDK v1.7.0_95 +1 checkstyle 0m 27s trunk passed +1 mvnsite 1m 27s trunk passed +1 mvneclipse 0m 26s trunk passed +1 findbugs 3m 33s trunk passed +1 javadoc 1m 24s trunk passed with JDK v1.8.0_72 +1 javadoc 2m 15s trunk passed with JDK v1.7.0_95 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 17s the patch passed +1 compile 1m 13s the patch passed with JDK v1.8.0_72 +1 javac 1m 13s the patch passed +1 compile 1m 20s the patch passed with JDK v1.7.0_95 +1 javac 1m 20s the patch passed +1 checkstyle 0m 24s the patch passed +1 mvnsite 1m 22s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 59s the patch passed +1 javadoc 1m 22s the patch passed with JDK v1.8.0_72 +1 javadoc 2m 7s the patch passed with JDK v1.7.0_95 +1 unit 0m 49s hadoop-hdfs-client in the patch passed with JDK v1.8.0_72. +1 unit 67m 50s hadoop-hdfs in the patch passed with JDK v1.8.0_72. +1 unit 0m 57s hadoop-hdfs-client in the patch passed with JDK v1.7.0_95. -1 unit 73m 33s hadoop-hdfs in the patch failed with JDK v1.7.0_95. +1 asflicense 0m 23s Patch does not generate ASF License warnings. 178m 53s Reason Tests JDK v1.7.0_95 Failed junit tests hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot JDK v1.7.0_95 Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12790889/HADOOP-12827.004.patch JIRA Issue HDFS-9887 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux cf8e35fbf01f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 27e0681 Default Java 1.7.0_95 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_72 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HDFS-Build/14687/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/14687/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_95.txt JDK v1.7.0_95 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/14687/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project Console output https://builds.apache.org/job/PreCommit-HDFS-Build/14687/console Powered by Apache Yetus 0.3.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        I think this patch is not complete – if for some reason SSL configuration throws an exception, the socket timeouts will not be configured, even if the connection is supposed to be webhdfs only, not swebhdfs.

        Show
        jojochuang Wei-Chiu Chuang added a comment - I think this patch is not complete – if for some reason SSL configuration throws an exception, the socket timeouts will not be configured, even if the connection is supposed to be webhdfs only, not swebhdfs.
        Hide
        xyao Xiaoyu Yao added a comment -

        Agree, this is a bug. Webhdfs with ssl configuration exception will not honor the configurable webhdfs connect/read timeout. It will always be DEFAULT_TIMEOUT_CONN_CONFIGURATOR the default value (1 min).

        Show
        xyao Xiaoyu Yao added a comment - Agree, this is a bug. Webhdfs with ssl configuration exception will not honor the configurable webhdfs connect/read timeout. It will always be DEFAULT_TIMEOUT_CONN_CONFIGURATOR the default value (1 min).
        Hide
        xyao Xiaoyu Yao added a comment -

        Thanks Wei-Chiu Chuang for reporting this. Further reading found that the webhdfs specific read/connect timeout implemented by HDFS-9887 should not affect other callers of URLConnectionFactory.newSslConnConfigurator() such as QuorumJournalManager#QuorumJournalManger(), DFSck#DFSck() and TransferFsImage(). I will file separate ticket to fix it.

        Show
        xyao Xiaoyu Yao added a comment - Thanks Wei-Chiu Chuang for reporting this. Further reading found that the webhdfs specific read/connect timeout implemented by HDFS-9887 should not affect other callers of URLConnectionFactory.newSslConnConfigurator() such as QuorumJournalManager#QuorumJournalManger(), DFSck#DFSck() and TransferFsImage() . I will file separate ticket to fix it.
        Hide
        xyao Xiaoyu Yao added a comment -

        Filed HDFS-9914 for the fix.

        Show
        xyao Xiaoyu Yao added a comment - Filed HDFS-9914 for the fix.

          People

          • Assignee:
            and1000 Austin Donnelly
            Reporter:
            and1000 Austin Donnelly
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development