Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.8.0, 3.0.0-alpha1
    • io, ipc
    • None
    • Reviewed

    Description

      In current Async DFS implementation, file system calls are invoked and returns Future<?> immediately to clients. Clients call Future#get to retrieve final results. Future#get internally invokes a chain of callbacks residing in ClientNamenodeProtocolTranslatorPB, ProtobufRpcEngine and ipc.Client. The callback path bypasses the original retry layer/logic designed for synchronous DFS. This proposes refactoring to make retry also works for Async DFS.

      Attachments

        1. h10433_20160528c.patch
          47 kB
          Tsz-wo Sze
        2. h10433_20160528.patch
          45 kB
          Tsz-wo Sze
        3. h10433_20160527.patch
          49 kB
          Tsz-wo Sze
        4. h10433_20160525b.patch
          37 kB
          Tsz-wo Sze
        5. h10433_20160525.patch
          37 kB
          Tsz-wo Sze
        6. h10433_20160524.patch
          31 kB
          Tsz-wo Sze

        Issue Links

          Activity

            szetszwo Tsz-wo Sze added a comment -

            h10433_20160524.patch: adds an async call queue to RetryInvocationHandler to support retry and failover.

            szetszwo Tsz-wo Sze added a comment - h10433_20160524.patch: adds an async call queue to RetryInvocationHandler to support retry and failover.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 15s Docker mode activated.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
            0 mvndep 0m 12s Maven dependency ordering for branch
            +1 mvninstall 6m 45s trunk passed
            +1 compile 7m 13s trunk passed
            +1 checkstyle 1m 37s trunk passed
            +1 mvnsite 2m 42s trunk passed
            +1 mvneclipse 0m 41s trunk passed
            +1 findbugs 4m 50s trunk passed
            +1 javadoc 2m 32s trunk passed
            0 mvndep 0m 12s Maven dependency ordering for patch
            +1 mvninstall 2m 16s the patch passed
            +1 compile 7m 26s the patch passed
            -1 javac 9m 54s root generated 1 new + 695 unchanged - 1 fixed = 696 total (was 696)
            +1 javac 7m 26s the patch passed
            -1 checkstyle 1m 34s root: patch generated 42 new + 211 unchanged - 5 fixed = 253 total (was 216)
            +1 mvnsite 2m 50s the patch passed
            +1 mvneclipse 0m 43s the patch passed
            +1 whitespace 0m 0s Patch has no whitespace issues.
            -1 findbugs 1m 44s hadoop-common-project/hadoop-common generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)
            +1 javadoc 2m 40s the patch passed
            +1 unit 9m 9s hadoop-common in the patch passed.
            +1 unit 0m 57s hadoop-hdfs-client in the patch passed.
            -1 unit 92m 6s hadoop-hdfs in the patch failed.
            +1 asflicense 0m 35s Patch does not generate ASF License warnings.
            153m 59s



            Reason Tests
            FindBugs module:hadoop-common-project/hadoop-common
              Inconsistent synchronization of org.apache.hadoop.io.retry.RetryInvocationHandler$CallReturn.returnValue; locked 66% of time Unsynchronized access at RetryInvocationHandler.java:66% of time Unsynchronized access at RetryInvocationHandler.java:[line 219]
              Inconsistent synchronization of org.apache.hadoop.io.retry.RetryInvocationHandler$CallReturn.state; locked 83% of time Unsynchronized access at RetryInvocationHandler.java:83% of time Unsynchronized access at RetryInvocationHandler.java:[line 219]
              Inconsistent synchronization of org.apache.hadoop.io.retry.RetryInvocationHandler$CallReturn.thrown; locked 80% of time Unsynchronized access at RetryInvocationHandler.java:80% of time Unsynchronized access at RetryInvocationHandler.java:[line 219]
              Wait not in loop in org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java:org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java:[line 56]
            Failed junit tests hadoop.hdfs.TestAsyncDFS
              hadoop.hdfs.server.datanode.TestFsDatasetCache
              hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
              hadoop.hdfs.server.namenode.TestEditLog
              hadoop.hdfs.server.namenode.TestFileTruncate
              hadoop.hdfs.TestAsyncDFSRename
            Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:2c91fd8
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806044/h10433_20160524.patch
            JIRA Issue HDFS-10433
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
            uname Linux 40ea4f791581 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / 09b866f
            Default Java 1.8.0_91
            findbugs v3.0.0
            javac root: https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/diff-compile-javac-root.txt
            checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/diff-checkstyle-root.txt
            findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
            unit https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
            unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
            Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15555/testReport/
            modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
            Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15555/console
            Powered by Apache Yetus 0.2.0 http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 6m 45s trunk passed +1 compile 7m 13s trunk passed +1 checkstyle 1m 37s trunk passed +1 mvnsite 2m 42s trunk passed +1 mvneclipse 0m 41s trunk passed +1 findbugs 4m 50s trunk passed +1 javadoc 2m 32s trunk passed 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 2m 16s the patch passed +1 compile 7m 26s the patch passed -1 javac 9m 54s root generated 1 new + 695 unchanged - 1 fixed = 696 total (was 696) +1 javac 7m 26s the patch passed -1 checkstyle 1m 34s root: patch generated 42 new + 211 unchanged - 5 fixed = 253 total (was 216) +1 mvnsite 2m 50s the patch passed +1 mvneclipse 0m 43s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 44s hadoop-common-project/hadoop-common generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) +1 javadoc 2m 40s the patch passed +1 unit 9m 9s hadoop-common in the patch passed. +1 unit 0m 57s hadoop-hdfs-client in the patch passed. -1 unit 92m 6s hadoop-hdfs in the patch failed. +1 asflicense 0m 35s Patch does not generate ASF License warnings. 153m 59s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   Inconsistent synchronization of org.apache.hadoop.io.retry.RetryInvocationHandler$CallReturn.returnValue; locked 66% of time Unsynchronized access at RetryInvocationHandler.java:66% of time Unsynchronized access at RetryInvocationHandler.java: [line 219]   Inconsistent synchronization of org.apache.hadoop.io.retry.RetryInvocationHandler$CallReturn.state; locked 83% of time Unsynchronized access at RetryInvocationHandler.java:83% of time Unsynchronized access at RetryInvocationHandler.java: [line 219]   Inconsistent synchronization of org.apache.hadoop.io.retry.RetryInvocationHandler$CallReturn.thrown; locked 80% of time Unsynchronized access at RetryInvocationHandler.java:80% of time Unsynchronized access at RetryInvocationHandler.java: [line 219]   Wait not in loop in org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java:org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java: [line 56] Failed junit tests hadoop.hdfs.TestAsyncDFS   hadoop.hdfs.server.datanode.TestFsDatasetCache   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestAsyncDFSRename Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806044/h10433_20160524.patch JIRA Issue HDFS-10433 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 40ea4f791581 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 09b866f Default Java 1.8.0_91 findbugs v3.0.0 javac root: https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15555/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15555/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
            szetszwo Tsz-wo Sze added a comment -

            h10433_20160525.patch: fixes test failures and warnings.

            szetszwo Tsz-wo Sze added a comment - h10433_20160525.patch: fixes test failures and warnings.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 20s Docker mode activated.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
            0 mvndep 0m 37s Maven dependency ordering for branch
            +1 mvninstall 6m 42s trunk passed
            +1 compile 6m 33s trunk passed
            +1 checkstyle 1m 26s trunk passed
            +1 mvnsite 2m 42s trunk passed
            +1 mvneclipse 0m 58s trunk passed
            +1 findbugs 4m 19s trunk passed
            +1 javadoc 2m 28s trunk passed
            0 mvndep 0m 11s Maven dependency ordering for patch
            +1 mvninstall 1m 58s the patch passed
            +1 compile 7m 18s the patch passed
            +1 javac 7m 18s the patch passed
            -1 checkstyle 1m 30s root: patch generated 30 new + 225 unchanged - 6 fixed = 255 total (was 231)
            +1 mvnsite 2m 39s the patch passed
            +1 mvneclipse 0m 34s the patch passed
            +1 whitespace 0m 0s Patch has no whitespace issues.
            +1 xml 0m 2s The patch has no ill-formed XML file.
            -1 findbugs 1m 40s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
            +1 javadoc 2m 27s the patch passed
            +1 unit 8m 52s hadoop-common in the patch passed.
            +1 unit 0m 58s hadoop-hdfs-client in the patch passed.
            -1 unit 66m 16s hadoop-hdfs in the patch failed.
            +1 asflicense 0m 26s Patch does not generate ASF License warnings.
            125m 44s



            Reason Tests
            FindBugs module:hadoop-common-project/hadoop-common
              Wait not in loop in org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java:org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java:[line 56]
            Failed junit tests hadoop.hdfs.TestAsyncDFSRename
              hadoop.hdfs.TestErasureCodeBenchmarkThroughput



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:2c91fd8
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806249/h10433_20160525.patch
            JIRA Issue HDFS-10433
            Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
            uname Linux 616d6bd3c87a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / 3c83cee
            Default Java 1.8.0_91
            findbugs v3.0.0
            checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15568/artifact/patchprocess/diff-checkstyle-root.txt
            findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15568/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
            unit https://builds.apache.org/job/PreCommit-HDFS-Build/15568/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
            unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15568/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
            Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15568/testReport/
            modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
            Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15568/console
            Powered by Apache Yetus 0.2.0 http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 37s Maven dependency ordering for branch +1 mvninstall 6m 42s trunk passed +1 compile 6m 33s trunk passed +1 checkstyle 1m 26s trunk passed +1 mvnsite 2m 42s trunk passed +1 mvneclipse 0m 58s trunk passed +1 findbugs 4m 19s trunk passed +1 javadoc 2m 28s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 58s the patch passed +1 compile 7m 18s the patch passed +1 javac 7m 18s the patch passed -1 checkstyle 1m 30s root: patch generated 30 new + 225 unchanged - 6 fixed = 255 total (was 231) +1 mvnsite 2m 39s the patch passed +1 mvneclipse 0m 34s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. -1 findbugs 1m 40s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 2m 27s the patch passed +1 unit 8m 52s hadoop-common in the patch passed. +1 unit 0m 58s hadoop-hdfs-client in the patch passed. -1 unit 66m 16s hadoop-hdfs in the patch failed. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 125m 44s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   Wait not in loop in org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java:org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java: [line 56] Failed junit tests hadoop.hdfs.TestAsyncDFSRename   hadoop.hdfs.TestErasureCodeBenchmarkThroughput Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806249/h10433_20160525.patch JIRA Issue HDFS-10433 Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 616d6bd3c87a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3c83cee Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15568/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15568/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/15568/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15568/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15568/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15568/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
            szetszwo Tsz-wo Sze added a comment -

            h10433_20160525b.patch: fixes more warnings.

            Note that the indentation checkstyle warnings are bogus.

            szetszwo Tsz-wo Sze added a comment - h10433_20160525b.patch: fixes more warnings. Note that the indentation checkstyle warnings are bogus.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 11s Docker mode activated.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
            0 mvndep 0m 11s Maven dependency ordering for branch
            +1 mvninstall 6m 21s trunk passed
            +1 compile 6m 25s trunk passed
            +1 checkstyle 1m 22s trunk passed
            +1 mvnsite 2m 15s trunk passed
            +1 mvneclipse 0m 35s trunk passed
            +1 findbugs 4m 22s trunk passed
            +1 javadoc 2m 21s trunk passed
            0 mvndep 0m 11s Maven dependency ordering for patch
            +1 mvninstall 1m 58s the patch passed
            +1 compile 6m 15s the patch passed
            +1 javac 6m 15s the patch passed
            -1 checkstyle 1m 22s root: patch generated 26 new + 224 unchanged - 6 fixed = 250 total (was 230)
            +1 mvnsite 2m 17s the patch passed
            +1 mvneclipse 0m 35s the patch passed
            +1 whitespace 0m 0s Patch has no whitespace issues.
            +1 xml 0m 2s The patch has no ill-formed XML file.
            +1 findbugs 4m 55s the patch passed
            +1 javadoc 2m 20s the patch passed
            +1 unit 7m 7s hadoop-common in the patch passed.
            +1 unit 0m 54s hadoop-hdfs-client in the patch passed.
            +1 unit 58m 2s hadoop-hdfs in the patch passed.
            +1 asflicense 0m 24s Patch does not generate ASF License warnings.
            111m 19s



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:2c91fd8
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806273/h10433_20160525b.patch
            JIRA Issue HDFS-10433
            Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
            uname Linux 595da50cdba9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / 013532a
            Default Java 1.8.0_91
            findbugs v3.0.0
            checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15572/artifact/patchprocess/diff-checkstyle-root.txt
            Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15572/testReport/
            modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
            Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15572/console
            Powered by Apache Yetus 0.2.0 http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 6m 21s trunk passed +1 compile 6m 25s trunk passed +1 checkstyle 1m 22s trunk passed +1 mvnsite 2m 15s trunk passed +1 mvneclipse 0m 35s trunk passed +1 findbugs 4m 22s trunk passed +1 javadoc 2m 21s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 58s the patch passed +1 compile 6m 15s the patch passed +1 javac 6m 15s the patch passed -1 checkstyle 1m 22s root: patch generated 26 new + 224 unchanged - 6 fixed = 250 total (was 230) +1 mvnsite 2m 17s the patch passed +1 mvneclipse 0m 35s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 4m 55s the patch passed +1 javadoc 2m 20s the patch passed +1 unit 7m 7s hadoop-common in the patch passed. +1 unit 0m 54s hadoop-hdfs-client in the patch passed. +1 unit 58m 2s hadoop-hdfs in the patch passed. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 111m 19s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806273/h10433_20160525b.patch JIRA Issue HDFS-10433 Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 595da50cdba9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 013532a Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15572/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15572/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15572/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
            szetszwo Tsz-wo Sze added a comment -

            Thanks Jing for the offline discussion. Here is a new patch.

            h10433_20160527.patch:

            • refactors async logic to AsyncCallHandler,
            • fixes a bug for the proxies without retry, and
            • add a new test, testAsyncWithoutRetry.
            szetszwo Tsz-wo Sze added a comment - Thanks Jing for the offline discussion. Here is a new patch. h10433_20160527.patch: refactors async logic to AsyncCallHandler, fixes a bug for the proxies without retry, and add a new test, testAsyncWithoutRetry.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 0s Docker mode activated.
            -1 patch 0m 5s HDFS-10433 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s HDFS-10433 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806785/h10433_20160527.patch JIRA Issue HDFS-10433 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15597/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
            szetszwo Tsz-wo Sze added a comment -

            h10433_20160528.patch: sync'ed with trunk.

            szetszwo Tsz-wo Sze added a comment - h10433_20160528.patch: sync'ed with trunk.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 17s Docker mode activated.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
            0 mvndep 0m 39s Maven dependency ordering for branch
            +1 mvninstall 6m 10s trunk passed
            +1 compile 6m 13s trunk passed
            +1 checkstyle 1m 22s trunk passed
            +1 mvnsite 2m 15s trunk passed
            +1 mvneclipse 0m 35s trunk passed
            +1 findbugs 4m 17s trunk passed
            +1 javadoc 2m 16s trunk passed
            0 mvndep 0m 12s Maven dependency ordering for patch
            +1 mvninstall 1m 52s the patch passed
            +1 compile 6m 11s the patch passed
            +1 javac 6m 11s the patch passed
            -1 checkstyle 1m 20s root: The patch generated 46 new + 224 unchanged - 6 fixed = 270 total (was 230)
            +1 mvnsite 2m 11s the patch passed
            +1 mvneclipse 0m 34s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 findbugs 4m 40s the patch passed
            +1 javadoc 2m 17s the patch passed
            +1 unit 12m 49s hadoop-common in the patch passed.
            +1 unit 0m 54s hadoop-hdfs-client in the patch passed.
            -1 unit 74m 14s hadoop-hdfs in the patch failed.
            +1 asflicense 0m 27s The patch does not generate ASF License warnings.
            132m 44s



            Reason Tests
            Failed junit tests hadoop.hdfs.TestErasureCodeBenchmarkThroughput



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:2c91fd8
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806827/h10433_20160528.patch
            JIRA Issue HDFS-10433
            Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
            uname Linux ebb0db7a1a3a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / f5ff05c
            Default Java 1.8.0_91
            findbugs v3.0.0
            checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15598/artifact/patchprocess/diff-checkstyle-root.txt
            unit https://builds.apache.org/job/PreCommit-HDFS-Build/15598/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
            unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15598/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
            Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15598/testReport/
            modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
            Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15598/console
            Powered by Apache Yetus 0.3.0 http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 39s Maven dependency ordering for branch +1 mvninstall 6m 10s trunk passed +1 compile 6m 13s trunk passed +1 checkstyle 1m 22s trunk passed +1 mvnsite 2m 15s trunk passed +1 mvneclipse 0m 35s trunk passed +1 findbugs 4m 17s trunk passed +1 javadoc 2m 16s trunk passed 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 52s the patch passed +1 compile 6m 11s the patch passed +1 javac 6m 11s the patch passed -1 checkstyle 1m 20s root: The patch generated 46 new + 224 unchanged - 6 fixed = 270 total (was 230) +1 mvnsite 2m 11s the patch passed +1 mvneclipse 0m 34s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 40s the patch passed +1 javadoc 2m 17s the patch passed +1 unit 12m 49s hadoop-common in the patch passed. +1 unit 0m 54s hadoop-hdfs-client in the patch passed. -1 unit 74m 14s hadoop-hdfs in the patch failed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 132m 44s Reason Tests Failed junit tests hadoop.hdfs.TestErasureCodeBenchmarkThroughput Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806827/h10433_20160528.patch JIRA Issue HDFS-10433 Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ebb0db7a1a3a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f5ff05c Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15598/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15598/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15598/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15598/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15598/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
            szetszwo Tsz-wo Sze added a comment -

            h10433_20160528b.patch: makes testAsyncWithoutRetry to be run in both HA and non-HA cases.

            szetszwo Tsz-wo Sze added a comment - h10433_20160528b.patch: makes testAsyncWithoutRetry to be run in both HA and non-HA cases.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 15s Docker mode activated.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
            0 mvndep 0m 48s Maven dependency ordering for branch
            +1 mvninstall 6m 15s trunk passed
            +1 compile 6m 14s trunk passed
            +1 checkstyle 1m 24s trunk passed
            +1 mvnsite 2m 30s trunk passed
            +1 mvneclipse 0m 35s trunk passed
            +1 findbugs 5m 16s trunk passed
            +1 javadoc 2m 29s trunk passed
            0 mvndep 0m 15s Maven dependency ordering for patch
            +1 mvninstall 2m 29s the patch passed
            +1 compile 8m 15s the patch passed
            +1 javac 8m 15s the patch passed
            -1 checkstyle 1m 22s root: The patch generated 44 new + 224 unchanged - 6 fixed = 268 total (was 230)
            +1 mvnsite 2m 12s the patch passed
            +1 mvneclipse 0m 35s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 findbugs 4m 50s the patch passed
            +1 javadoc 2m 21s the patch passed
            +1 unit 9m 28s hadoop-common in the patch passed.
            +1 unit 1m 8s hadoop-hdfs-client in the patch passed.
            -1 unit 76m 30s hadoop-hdfs in the patch failed.
            +1 asflicense 0m 22s The patch does not generate ASF License warnings.
            136m 37s



            Reason Tests
            Failed junit tests hadoop.hdfs.TestPersistBlocks
              hadoop.hdfs.TestAsyncHDFSWithHA



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:2c91fd8
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806837/h10433_20160528b.patch
            JIRA Issue HDFS-10433
            Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
            uname Linux 990848a97abf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / f5ff05c
            Default Java 1.8.0_91
            findbugs v3.0.0
            checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15599/artifact/patchprocess/diff-checkstyle-root.txt
            unit https://builds.apache.org/job/PreCommit-HDFS-Build/15599/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
            unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15599/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
            Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15599/testReport/
            modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
            Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15599/console
            Powered by Apache Yetus 0.3.0 http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 48s Maven dependency ordering for branch +1 mvninstall 6m 15s trunk passed +1 compile 6m 14s trunk passed +1 checkstyle 1m 24s trunk passed +1 mvnsite 2m 30s trunk passed +1 mvneclipse 0m 35s trunk passed +1 findbugs 5m 16s trunk passed +1 javadoc 2m 29s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 29s the patch passed +1 compile 8m 15s the patch passed +1 javac 8m 15s the patch passed -1 checkstyle 1m 22s root: The patch generated 44 new + 224 unchanged - 6 fixed = 268 total (was 230) +1 mvnsite 2m 12s the patch passed +1 mvneclipse 0m 35s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 50s the patch passed +1 javadoc 2m 21s the patch passed +1 unit 9m 28s hadoop-common in the patch passed. +1 unit 1m 8s hadoop-hdfs-client in the patch passed. -1 unit 76m 30s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 136m 37s Reason Tests Failed junit tests hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.TestAsyncHDFSWithHA Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806837/h10433_20160528b.patch JIRA Issue HDFS-10433 Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 990848a97abf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f5ff05c Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15599/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15599/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15599/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15599/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15599/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
            szetszwo Tsz-wo Sze added a comment -

            h10433_20160528c.patch: fixes test failure.

            szetszwo Tsz-wo Sze added a comment - h10433_20160528c.patch: fixes test failure.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 14s Docker mode activated.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
            0 mvndep 0m 17s Maven dependency ordering for branch
            +1 mvninstall 6m 25s trunk passed
            +1 compile 8m 22s trunk passed
            +1 checkstyle 1m 23s trunk passed
            +1 mvnsite 2m 55s trunk passed
            +1 mvneclipse 0m 39s trunk passed
            +1 findbugs 5m 32s trunk passed
            +1 javadoc 2m 19s trunk passed
            0 mvndep 0m 11s Maven dependency ordering for patch
            +1 mvninstall 1m 53s the patch passed
            +1 compile 6m 36s the patch passed
            +1 javac 6m 36s the patch passed
            -1 checkstyle 1m 22s root: The patch generated 44 new + 224 unchanged - 6 fixed = 268 total (was 230)
            +1 mvnsite 2m 51s the patch passed
            +1 mvneclipse 0m 42s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 1s The patch has no ill-formed XML file.
            +1 findbugs 5m 37s the patch passed
            +1 javadoc 2m 44s the patch passed
            +1 unit 8m 41s hadoop-common in the patch passed.
            +1 unit 0m 57s hadoop-hdfs-client in the patch passed.
            -1 unit 75m 47s hadoop-hdfs in the patch failed.
            +1 asflicense 0m 22s The patch does not generate ASF License warnings.
            136m 50s



            Reason Tests
            Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache
              hadoop.hdfs.TestAsyncHDFSWithHA
              hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:2c91fd8
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806837/h10433_20160528b.patch
            JIRA Issue HDFS-10433
            Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
            uname Linux b56bdee95ca5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / f5ff05c
            Default Java 1.8.0_91
            findbugs v3.0.0
            checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15600/artifact/patchprocess/diff-checkstyle-root.txt
            unit https://builds.apache.org/job/PreCommit-HDFS-Build/15600/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
            unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15600/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
            Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15600/testReport/
            modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
            Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15600/console
            Powered by Apache Yetus 0.3.0 http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 6m 25s trunk passed +1 compile 8m 22s trunk passed +1 checkstyle 1m 23s trunk passed +1 mvnsite 2m 55s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 5m 32s trunk passed +1 javadoc 2m 19s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 53s the patch passed +1 compile 6m 36s the patch passed +1 javac 6m 36s the patch passed -1 checkstyle 1m 22s root: The patch generated 44 new + 224 unchanged - 6 fixed = 268 total (was 230) +1 mvnsite 2m 51s the patch passed +1 mvneclipse 0m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 5m 37s the patch passed +1 javadoc 2m 44s the patch passed +1 unit 8m 41s hadoop-common in the patch passed. +1 unit 0m 57s hadoop-hdfs-client in the patch passed. -1 unit 75m 47s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 136m 50s Reason Tests Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestAsyncHDFSWithHA   hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806837/h10433_20160528b.patch JIRA Issue HDFS-10433 Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b56bdee95ca5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f5ff05c Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15600/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15600/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15600/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15600/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15600/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
            szetszwo Tsz-wo Sze added a comment -

            Jenkins somehow tested h10433_20160528b.patch again but not /h10433_20160528c.patch.

            szetszwo Tsz-wo Sze added a comment - Jenkins somehow tested h10433_20160528b.patch again but not /h10433_20160528c.patch.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 16s Docker mode activated.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
            0 mvndep 0m 11s Maven dependency ordering for branch
            +1 mvninstall 6m 22s trunk passed
            +1 compile 8m 20s trunk passed
            +1 checkstyle 1m 25s trunk passed
            +1 mvnsite 2m 52s trunk passed
            +1 mvneclipse 0m 40s trunk passed
            +1 findbugs 5m 36s trunk passed
            +1 javadoc 2m 18s trunk passed
            0 mvndep 0m 12s Maven dependency ordering for patch
            +1 mvninstall 1m 51s the patch passed
            +1 compile 6m 31s the patch passed
            +1 javac 6m 31s the patch passed
            -1 checkstyle 1m 22s root: The patch generated 44 new + 224 unchanged - 6 fixed = 268 total (was 230)
            +1 mvnsite 2m 52s the patch passed
            +1 mvneclipse 0m 40s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 xml 0m 2s The patch has no ill-formed XML file.
            +1 findbugs 5m 38s the patch passed
            +1 javadoc 2m 48s the patch passed
            +1 unit 8m 46s hadoop-common in the patch passed.
            +1 unit 0m 52s hadoop-hdfs-client in the patch passed.
            -1 unit 72m 58s hadoop-hdfs in the patch failed.
            +1 asflicense 0m 24s The patch does not generate ASF License warnings.
            133m 58s



            Reason Tests
            Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
              hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
              hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:2c91fd8
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806846/h10433_20160528c.patch
            JIRA Issue HDFS-10433
            Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
            uname Linux dd63698998c9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / 4e1f56e
            Default Java 1.8.0_91
            findbugs v3.0.0
            checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15602/artifact/patchprocess/diff-checkstyle-root.txt
            unit https://builds.apache.org/job/PreCommit-HDFS-Build/15602/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
            unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15602/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
            Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15602/testReport/
            modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
            Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15602/console
            Powered by Apache Yetus 0.3.0 http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 6m 22s trunk passed +1 compile 8m 20s trunk passed +1 checkstyle 1m 25s trunk passed +1 mvnsite 2m 52s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 5m 36s trunk passed +1 javadoc 2m 18s trunk passed 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 51s the patch passed +1 compile 6m 31s the patch passed +1 javac 6m 31s the patch passed -1 checkstyle 1m 22s root: The patch generated 44 new + 224 unchanged - 6 fixed = 268 total (was 230) +1 mvnsite 2m 52s the patch passed +1 mvneclipse 0m 40s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 5m 38s the patch passed +1 javadoc 2m 48s the patch passed +1 unit 8m 46s hadoop-common in the patch passed. +1 unit 0m 52s hadoop-hdfs-client in the patch passed. -1 unit 72m 58s hadoop-hdfs in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 133m 58s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.TestReconstructStripedBlocks   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806846/h10433_20160528c.patch JIRA Issue HDFS-10433 Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux dd63698998c9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4e1f56e Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15602/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15602/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15602/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15602/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15602/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
            jingzhao Jing Zhao added a comment -

            Thanks for updating the patch, szetszwo! The latest patch looks good to me overall. Two questions:

            1. The interval between two retries is realized by Thread.sleep, which makes the background thread in AsyncCallHandler sleep. Because all the client side Future.get calls need to wait until the background thread for the final result, this sleep may delay all the pending requests.
            2. The current background thread does a sleep inside of the loop, which may delay all the RPC requests. Ideally we want this thread to wait for response notification from RPC client.
            3. Minor: Counters can be created inside of the Call's constructor method instead of being passed as a parameter.

            Looks like #1 and #2 need some extra work. Considering the current patch is already complicated, we can address them in a separate jira. I will give +1 for committing the current patch first.

            jingzhao Jing Zhao added a comment - Thanks for updating the patch, szetszwo ! The latest patch looks good to me overall. Two questions: The interval between two retries is realized by Thread.sleep , which makes the background thread in AsyncCallHandler sleep. Because all the client side Future.get calls need to wait until the background thread for the final result, this sleep may delay all the pending requests. The current background thread does a sleep inside of the loop, which may delay all the RPC requests. Ideally we want this thread to wait for response notification from RPC client. Minor: Counters can be created inside of the Call's constructor method instead of being passed as a parameter. Looks like #1 and #2 need some extra work. Considering the current patch is already complicated, we can address them in a separate jira. I will give +1 for committing the current patch first.
            szetszwo Tsz-wo Sze added a comment -

            Thanks Jing. I will commit this shortly and file a follow up JIRA.

            szetszwo Tsz-wo Sze added a comment - Thanks Jing. I will commit this shortly and file a follow up JIRA.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 0s Docker mode activated.
            -1 patch 0m 5s HADOOP-13226 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s HADOOP-13226 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806846/h10433_20160528c.patch JIRA Issue HADOOP-13226 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9632/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            hudson Hudson added a comment -

            SUCCESS: Integrated in Hadoop-trunk-Commit #9889 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9889/)
            HADOOP-13226 Support async call retry and failover. (szetszwo: rev 83f2f78c118a7e52aba5104bd97b0acedc96be7b)

            • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
            • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestAsyncIPC.java
            • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/AsyncDistributedFileSystem.java
            • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
            • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncHDFSWithHA.java
            • hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
            • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/AsyncCallHandler.java
            • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/CallReturn.java
            • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
            • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
            • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncDFS.java
            • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java
            • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/concurrent/AsyncGet.java
            • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java
            hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9889 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9889/ ) HADOOP-13226 Support async call retry and failover. (szetszwo: rev 83f2f78c118a7e52aba5104bd97b0acedc96be7b) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestAsyncIPC.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/AsyncDistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncHDFSWithHA.java hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/AsyncCallHandler.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/CallReturn.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncDFS.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/concurrent/AsyncGet.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java
            szetszwo Tsz-wo Sze added a comment -

            Thanks Jing for reviewing the patches and the great ideas!

            I have committed this.

            szetszwo Tsz-wo Sze added a comment - Thanks Jing for reviewing the patches and the great ideas! I have committed this.
            hudson Hudson added a comment -

            SUCCESS: Integrated in Hadoop-trunk-Commit #9913 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9913/)
            Revert "HADOOP-13226 Support async call retry and failover." (wang: rev 5360da8bd9f720384860f411bee081aef13b4bd4)

            • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/AsyncCallHandler.java
            • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java
            • hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
            • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/CallReturn.java
            • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
            • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestAsyncIPC.java
            • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/AsyncDistributedFileSystem.java
            • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncDFS.java
            • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncHDFSWithHA.java
            • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java
            • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
            • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/concurrent/AsyncGet.java
            • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
            • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
            hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9913 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9913/ ) Revert " HADOOP-13226 Support async call retry and failover." (wang: rev 5360da8bd9f720384860f411bee081aef13b4bd4) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/AsyncCallHandler.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/CallReturn.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestAsyncIPC.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/AsyncDistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncDFS.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncHDFSWithHA.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/concurrent/AsyncGet.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java

            People

              szetszwo Tsz-wo Sze
              xiaobingo Xiaobing Zhou
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: