Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: io, ipc
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In current Async DFS implementation, file system calls are invoked and returns Future<?> immediately to clients. Clients call Future#get to retrieve final results. Future#get internally invokes a chain of callbacks residing in ClientNamenodeProtocolTranslatorPB, ProtobufRpcEngine and ipc.Client. The callback path bypasses the original retry layer/logic designed for synchronous DFS. This proposes refactoring to make retry also works for Async DFS.

      1. h10433_20160524.patch
        31 kB
        Tsz Wo Nicholas Sze
      2. h10433_20160525.patch
        37 kB
        Tsz Wo Nicholas Sze
      3. h10433_20160525b.patch
        37 kB
        Tsz Wo Nicholas Sze
      4. h10433_20160527.patch
        49 kB
        Tsz Wo Nicholas Sze
      5. h10433_20160528.patch
        45 kB
        Tsz Wo Nicholas Sze
      6. h10433_20160528c.patch
        47 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h10433_20160524.patch: adds an async call queue to RetryInvocationHandler to support retry and failover.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h10433_20160524.patch: adds an async call queue to RetryInvocationHandler to support retry and failover.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 12s Maven dependency ordering for branch
          +1 mvninstall 6m 45s trunk passed
          +1 compile 7m 13s trunk passed
          +1 checkstyle 1m 37s trunk passed
          +1 mvnsite 2m 42s trunk passed
          +1 mvneclipse 0m 41s trunk passed
          +1 findbugs 4m 50s trunk passed
          +1 javadoc 2m 32s trunk passed
          0 mvndep 0m 12s Maven dependency ordering for patch
          +1 mvninstall 2m 16s the patch passed
          +1 compile 7m 26s the patch passed
          -1 javac 9m 54s root generated 1 new + 695 unchanged - 1 fixed = 696 total (was 696)
          +1 javac 7m 26s the patch passed
          -1 checkstyle 1m 34s root: patch generated 42 new + 211 unchanged - 5 fixed = 253 total (was 216)
          +1 mvnsite 2m 50s the patch passed
          +1 mvneclipse 0m 43s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          -1 findbugs 1m 44s hadoop-common-project/hadoop-common generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0)
          +1 javadoc 2m 40s the patch passed
          +1 unit 9m 9s hadoop-common in the patch passed.
          +1 unit 0m 57s hadoop-hdfs-client in the patch passed.
          -1 unit 92m 6s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 35s Patch does not generate ASF License warnings.
          153m 59s



          Reason Tests
          FindBugs module:hadoop-common-project/hadoop-common
            Inconsistent synchronization of org.apache.hadoop.io.retry.RetryInvocationHandler$CallReturn.returnValue; locked 66% of time Unsynchronized access at RetryInvocationHandler.java:66% of time Unsynchronized access at RetryInvocationHandler.java:[line 219]
            Inconsistent synchronization of org.apache.hadoop.io.retry.RetryInvocationHandler$CallReturn.state; locked 83% of time Unsynchronized access at RetryInvocationHandler.java:83% of time Unsynchronized access at RetryInvocationHandler.java:[line 219]
            Inconsistent synchronization of org.apache.hadoop.io.retry.RetryInvocationHandler$CallReturn.thrown; locked 80% of time Unsynchronized access at RetryInvocationHandler.java:80% of time Unsynchronized access at RetryInvocationHandler.java:[line 219]
            Wait not in loop in org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java:org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java:[line 56]
          Failed junit tests hadoop.hdfs.TestAsyncDFS
            hadoop.hdfs.server.datanode.TestFsDatasetCache
            hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.server.namenode.TestEditLog
            hadoop.hdfs.server.namenode.TestFileTruncate
            hadoop.hdfs.TestAsyncDFSRename
          Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806044/h10433_20160524.patch
          JIRA Issue HDFS-10433
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 40ea4f791581 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 09b866f
          Default Java 1.8.0_91
          findbugs v3.0.0
          javac root: https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/diff-compile-javac-root.txt
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15555/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15555/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 6m 45s trunk passed +1 compile 7m 13s trunk passed +1 checkstyle 1m 37s trunk passed +1 mvnsite 2m 42s trunk passed +1 mvneclipse 0m 41s trunk passed +1 findbugs 4m 50s trunk passed +1 javadoc 2m 32s trunk passed 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 2m 16s the patch passed +1 compile 7m 26s the patch passed -1 javac 9m 54s root generated 1 new + 695 unchanged - 1 fixed = 696 total (was 696) +1 javac 7m 26s the patch passed -1 checkstyle 1m 34s root: patch generated 42 new + 211 unchanged - 5 fixed = 253 total (was 216) +1 mvnsite 2m 50s the patch passed +1 mvneclipse 0m 43s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. -1 findbugs 1m 44s hadoop-common-project/hadoop-common generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) +1 javadoc 2m 40s the patch passed +1 unit 9m 9s hadoop-common in the patch passed. +1 unit 0m 57s hadoop-hdfs-client in the patch passed. -1 unit 92m 6s hadoop-hdfs in the patch failed. +1 asflicense 0m 35s Patch does not generate ASF License warnings. 153m 59s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   Inconsistent synchronization of org.apache.hadoop.io.retry.RetryInvocationHandler$CallReturn.returnValue; locked 66% of time Unsynchronized access at RetryInvocationHandler.java:66% of time Unsynchronized access at RetryInvocationHandler.java: [line 219]   Inconsistent synchronization of org.apache.hadoop.io.retry.RetryInvocationHandler$CallReturn.state; locked 83% of time Unsynchronized access at RetryInvocationHandler.java:83% of time Unsynchronized access at RetryInvocationHandler.java: [line 219]   Inconsistent synchronization of org.apache.hadoop.io.retry.RetryInvocationHandler$CallReturn.thrown; locked 80% of time Unsynchronized access at RetryInvocationHandler.java:80% of time Unsynchronized access at RetryInvocationHandler.java: [line 219]   Wait not in loop in org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java:org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java: [line 56] Failed junit tests hadoop.hdfs.TestAsyncDFS   hadoop.hdfs.server.datanode.TestFsDatasetCache   hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.namenode.TestEditLog   hadoop.hdfs.server.namenode.TestFileTruncate   hadoop.hdfs.TestAsyncDFSRename Timed out junit tests org.apache.hadoop.hdfs.TestLeaseRecovery2 Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806044/h10433_20160524.patch JIRA Issue HDFS-10433 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 40ea4f791581 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 09b866f Default Java 1.8.0_91 findbugs v3.0.0 javac root: https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15555/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15555/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15555/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h10433_20160525.patch: fixes test failures and warnings.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h10433_20160525.patch: fixes test failures and warnings.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 37s Maven dependency ordering for branch
          +1 mvninstall 6m 42s trunk passed
          +1 compile 6m 33s trunk passed
          +1 checkstyle 1m 26s trunk passed
          +1 mvnsite 2m 42s trunk passed
          +1 mvneclipse 0m 58s trunk passed
          +1 findbugs 4m 19s trunk passed
          +1 javadoc 2m 28s trunk passed
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 1m 58s the patch passed
          +1 compile 7m 18s the patch passed
          +1 javac 7m 18s the patch passed
          -1 checkstyle 1m 30s root: patch generated 30 new + 225 unchanged - 6 fixed = 255 total (was 231)
          +1 mvnsite 2m 39s the patch passed
          +1 mvneclipse 0m 34s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          -1 findbugs 1m 40s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javadoc 2m 27s the patch passed
          +1 unit 8m 52s hadoop-common in the patch passed.
          +1 unit 0m 58s hadoop-hdfs-client in the patch passed.
          -1 unit 66m 16s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 26s Patch does not generate ASF License warnings.
          125m 44s



          Reason Tests
          FindBugs module:hadoop-common-project/hadoop-common
            Wait not in loop in org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java:org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java:[line 56]
          Failed junit tests hadoop.hdfs.TestAsyncDFSRename
            hadoop.hdfs.TestErasureCodeBenchmarkThroughput



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806249/h10433_20160525.patch
          JIRA Issue HDFS-10433
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 616d6bd3c87a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3c83cee
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15568/artifact/patchprocess/diff-checkstyle-root.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15568/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15568/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15568/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15568/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15568/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 37s Maven dependency ordering for branch +1 mvninstall 6m 42s trunk passed +1 compile 6m 33s trunk passed +1 checkstyle 1m 26s trunk passed +1 mvnsite 2m 42s trunk passed +1 mvneclipse 0m 58s trunk passed +1 findbugs 4m 19s trunk passed +1 javadoc 2m 28s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 58s the patch passed +1 compile 7m 18s the patch passed +1 javac 7m 18s the patch passed -1 checkstyle 1m 30s root: patch generated 30 new + 225 unchanged - 6 fixed = 255 total (was 231) +1 mvnsite 2m 39s the patch passed +1 mvneclipse 0m 34s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. -1 findbugs 1m 40s hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javadoc 2m 27s the patch passed +1 unit 8m 52s hadoop-common in the patch passed. +1 unit 0m 58s hadoop-hdfs-client in the patch passed. -1 unit 66m 16s hadoop-hdfs in the patch failed. +1 asflicense 0m 26s Patch does not generate ASF License warnings. 125m 44s Reason Tests FindBugs module:hadoop-common-project/hadoop-common   Wait not in loop in org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java:org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java: [line 56] Failed junit tests hadoop.hdfs.TestAsyncDFSRename   hadoop.hdfs.TestErasureCodeBenchmarkThroughput Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806249/h10433_20160525.patch JIRA Issue HDFS-10433 Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 616d6bd3c87a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3c83cee Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15568/artifact/patchprocess/diff-checkstyle-root.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/15568/artifact/patchprocess/new-findbugs-hadoop-common-project_hadoop-common.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/15568/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15568/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15568/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15568/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h10433_20160525b.patch: fixes more warnings.

          Note that the indentation checkstyle warnings are bogus.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h10433_20160525b.patch: fixes more warnings. Note that the indentation checkstyle warnings are bogus.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 6m 21s trunk passed
          +1 compile 6m 25s trunk passed
          +1 checkstyle 1m 22s trunk passed
          +1 mvnsite 2m 15s trunk passed
          +1 mvneclipse 0m 35s trunk passed
          +1 findbugs 4m 22s trunk passed
          +1 javadoc 2m 21s trunk passed
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 1m 58s the patch passed
          +1 compile 6m 15s the patch passed
          +1 javac 6m 15s the patch passed
          -1 checkstyle 1m 22s root: patch generated 26 new + 224 unchanged - 6 fixed = 250 total (was 230)
          +1 mvnsite 2m 17s the patch passed
          +1 mvneclipse 0m 35s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 4m 55s the patch passed
          +1 javadoc 2m 20s the patch passed
          +1 unit 7m 7s hadoop-common in the patch passed.
          +1 unit 0m 54s hadoop-hdfs-client in the patch passed.
          +1 unit 58m 2s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 24s Patch does not generate ASF License warnings.
          111m 19s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806273/h10433_20160525b.patch
          JIRA Issue HDFS-10433
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 595da50cdba9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 013532a
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15572/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15572/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15572/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 6m 21s trunk passed +1 compile 6m 25s trunk passed +1 checkstyle 1m 22s trunk passed +1 mvnsite 2m 15s trunk passed +1 mvneclipse 0m 35s trunk passed +1 findbugs 4m 22s trunk passed +1 javadoc 2m 21s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 58s the patch passed +1 compile 6m 15s the patch passed +1 javac 6m 15s the patch passed -1 checkstyle 1m 22s root: patch generated 26 new + 224 unchanged - 6 fixed = 250 total (was 230) +1 mvnsite 2m 17s the patch passed +1 mvneclipse 0m 35s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 4m 55s the patch passed +1 javadoc 2m 20s the patch passed +1 unit 7m 7s hadoop-common in the patch passed. +1 unit 0m 54s hadoop-hdfs-client in the patch passed. +1 unit 58m 2s hadoop-hdfs in the patch passed. +1 asflicense 0m 24s Patch does not generate ASF License warnings. 111m 19s Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806273/h10433_20160525b.patch JIRA Issue HDFS-10433 Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 595da50cdba9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 013532a Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15572/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15572/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15572/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thanks Jing for the offline discussion. Here is a new patch.

          h10433_20160527.patch:

          • refactors async logic to AsyncCallHandler,
          • fixes a bug for the proxies without retry, and
          • add a new test, testAsyncWithoutRetry.
          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thanks Jing for the offline discussion. Here is a new patch. h10433_20160527.patch: refactors async logic to AsyncCallHandler, fixes a bug for the proxies without retry, and add a new test, testAsyncWithoutRetry.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 5s HDFS-10433 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806785/h10433_20160527.patch
          JIRA Issue HDFS-10433
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15597/console
          Powered by Apache Yetus 0.2.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s HDFS-10433 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806785/h10433_20160527.patch JIRA Issue HDFS-10433 Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15597/console Powered by Apache Yetus 0.2.0 http://yetus.apache.org This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h10433_20160528.patch: sync'ed with trunk.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h10433_20160528.patch: sync'ed with trunk.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 39s Maven dependency ordering for branch
          +1 mvninstall 6m 10s trunk passed
          +1 compile 6m 13s trunk passed
          +1 checkstyle 1m 22s trunk passed
          +1 mvnsite 2m 15s trunk passed
          +1 mvneclipse 0m 35s trunk passed
          +1 findbugs 4m 17s trunk passed
          +1 javadoc 2m 16s trunk passed
          0 mvndep 0m 12s Maven dependency ordering for patch
          +1 mvninstall 1m 52s the patch passed
          +1 compile 6m 11s the patch passed
          +1 javac 6m 11s the patch passed
          -1 checkstyle 1m 20s root: The patch generated 46 new + 224 unchanged - 6 fixed = 270 total (was 230)
          +1 mvnsite 2m 11s the patch passed
          +1 mvneclipse 0m 34s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 4m 40s the patch passed
          +1 javadoc 2m 17s the patch passed
          +1 unit 12m 49s hadoop-common in the patch passed.
          +1 unit 0m 54s hadoop-hdfs-client in the patch passed.
          -1 unit 74m 14s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 27s The patch does not generate ASF License warnings.
          132m 44s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestErasureCodeBenchmarkThroughput



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806827/h10433_20160528.patch
          JIRA Issue HDFS-10433
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ebb0db7a1a3a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f5ff05c
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15598/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15598/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15598/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15598/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15598/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 39s Maven dependency ordering for branch +1 mvninstall 6m 10s trunk passed +1 compile 6m 13s trunk passed +1 checkstyle 1m 22s trunk passed +1 mvnsite 2m 15s trunk passed +1 mvneclipse 0m 35s trunk passed +1 findbugs 4m 17s trunk passed +1 javadoc 2m 16s trunk passed 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 52s the patch passed +1 compile 6m 11s the patch passed +1 javac 6m 11s the patch passed -1 checkstyle 1m 20s root: The patch generated 46 new + 224 unchanged - 6 fixed = 270 total (was 230) +1 mvnsite 2m 11s the patch passed +1 mvneclipse 0m 34s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 40s the patch passed +1 javadoc 2m 17s the patch passed +1 unit 12m 49s hadoop-common in the patch passed. +1 unit 0m 54s hadoop-hdfs-client in the patch passed. -1 unit 74m 14s hadoop-hdfs in the patch failed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 132m 44s Reason Tests Failed junit tests hadoop.hdfs.TestErasureCodeBenchmarkThroughput Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806827/h10433_20160528.patch JIRA Issue HDFS-10433 Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ebb0db7a1a3a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f5ff05c Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15598/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15598/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15598/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15598/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15598/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h10433_20160528b.patch: makes testAsyncWithoutRetry to be run in both HA and non-HA cases.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h10433_20160528b.patch: makes testAsyncWithoutRetry to be run in both HA and non-HA cases.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 15s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 48s Maven dependency ordering for branch
          +1 mvninstall 6m 15s trunk passed
          +1 compile 6m 14s trunk passed
          +1 checkstyle 1m 24s trunk passed
          +1 mvnsite 2m 30s trunk passed
          +1 mvneclipse 0m 35s trunk passed
          +1 findbugs 5m 16s trunk passed
          +1 javadoc 2m 29s trunk passed
          0 mvndep 0m 15s Maven dependency ordering for patch
          +1 mvninstall 2m 29s the patch passed
          +1 compile 8m 15s the patch passed
          +1 javac 8m 15s the patch passed
          -1 checkstyle 1m 22s root: The patch generated 44 new + 224 unchanged - 6 fixed = 268 total (was 230)
          +1 mvnsite 2m 12s the patch passed
          +1 mvneclipse 0m 35s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 4m 50s the patch passed
          +1 javadoc 2m 21s the patch passed
          +1 unit 9m 28s hadoop-common in the patch passed.
          +1 unit 1m 8s hadoop-hdfs-client in the patch passed.
          -1 unit 76m 30s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          136m 37s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestPersistBlocks
            hadoop.hdfs.TestAsyncHDFSWithHA



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806837/h10433_20160528b.patch
          JIRA Issue HDFS-10433
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 990848a97abf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f5ff05c
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15599/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15599/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15599/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15599/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15599/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 48s Maven dependency ordering for branch +1 mvninstall 6m 15s trunk passed +1 compile 6m 14s trunk passed +1 checkstyle 1m 24s trunk passed +1 mvnsite 2m 30s trunk passed +1 mvneclipse 0m 35s trunk passed +1 findbugs 5m 16s trunk passed +1 javadoc 2m 29s trunk passed 0 mvndep 0m 15s Maven dependency ordering for patch +1 mvninstall 2m 29s the patch passed +1 compile 8m 15s the patch passed +1 javac 8m 15s the patch passed -1 checkstyle 1m 22s root: The patch generated 44 new + 224 unchanged - 6 fixed = 268 total (was 230) +1 mvnsite 2m 12s the patch passed +1 mvneclipse 0m 35s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 50s the patch passed +1 javadoc 2m 21s the patch passed +1 unit 9m 28s hadoop-common in the patch passed. +1 unit 1m 8s hadoop-hdfs-client in the patch passed. -1 unit 76m 30s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 136m 37s Reason Tests Failed junit tests hadoop.hdfs.TestPersistBlocks   hadoop.hdfs.TestAsyncHDFSWithHA Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806837/h10433_20160528b.patch JIRA Issue HDFS-10433 Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 990848a97abf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f5ff05c Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15599/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15599/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15599/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15599/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15599/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          h10433_20160528c.patch: fixes test failure.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - h10433_20160528c.patch: fixes test failure.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 17s Maven dependency ordering for branch
          +1 mvninstall 6m 25s trunk passed
          +1 compile 8m 22s trunk passed
          +1 checkstyle 1m 23s trunk passed
          +1 mvnsite 2m 55s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 5m 32s trunk passed
          +1 javadoc 2m 19s trunk passed
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 1m 53s the patch passed
          +1 compile 6m 36s the patch passed
          +1 javac 6m 36s the patch passed
          -1 checkstyle 1m 22s root: The patch generated 44 new + 224 unchanged - 6 fixed = 268 total (was 230)
          +1 mvnsite 2m 51s the patch passed
          +1 mvneclipse 0m 42s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 5m 37s the patch passed
          +1 javadoc 2m 44s the patch passed
          +1 unit 8m 41s hadoop-common in the patch passed.
          +1 unit 0m 57s hadoop-hdfs-client in the patch passed.
          -1 unit 75m 47s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          136m 50s



          Reason Tests
          Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache
            hadoop.hdfs.TestAsyncHDFSWithHA
            hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806837/h10433_20160528b.patch
          JIRA Issue HDFS-10433
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux b56bdee95ca5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f5ff05c
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15600/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15600/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15600/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15600/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15600/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 17s Maven dependency ordering for branch +1 mvninstall 6m 25s trunk passed +1 compile 8m 22s trunk passed +1 checkstyle 1m 23s trunk passed +1 mvnsite 2m 55s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 5m 32s trunk passed +1 javadoc 2m 19s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 53s the patch passed +1 compile 6m 36s the patch passed +1 javac 6m 36s the patch passed -1 checkstyle 1m 22s root: The patch generated 44 new + 224 unchanged - 6 fixed = 268 total (was 230) +1 mvnsite 2m 51s the patch passed +1 mvneclipse 0m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 5m 37s the patch passed +1 javadoc 2m 44s the patch passed +1 unit 8m 41s hadoop-common in the patch passed. +1 unit 0m 57s hadoop-hdfs-client in the patch passed. -1 unit 75m 47s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 136m 50s Reason Tests Failed junit tests hadoop.hdfs.shortcircuit.TestShortCircuitCache   hadoop.hdfs.TestAsyncHDFSWithHA   hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806837/h10433_20160528b.patch JIRA Issue HDFS-10433 Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux b56bdee95ca5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f5ff05c Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15600/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15600/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15600/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15600/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15600/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Jenkins somehow tested h10433_20160528b.patch again but not /h10433_20160528c.patch.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Jenkins somehow tested h10433_20160528b.patch again but not /h10433_20160528c.patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 6m 22s trunk passed
          +1 compile 8m 20s trunk passed
          +1 checkstyle 1m 25s trunk passed
          +1 mvnsite 2m 52s trunk passed
          +1 mvneclipse 0m 40s trunk passed
          +1 findbugs 5m 36s trunk passed
          +1 javadoc 2m 18s trunk passed
          0 mvndep 0m 12s Maven dependency ordering for patch
          +1 mvninstall 1m 51s the patch passed
          +1 compile 6m 31s the patch passed
          +1 javac 6m 31s the patch passed
          -1 checkstyle 1m 22s root: The patch generated 44 new + 224 unchanged - 6 fixed = 268 total (was 230)
          +1 mvnsite 2m 52s the patch passed
          +1 mvneclipse 0m 40s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          +1 findbugs 5m 38s the patch passed
          +1 javadoc 2m 48s the patch passed
          +1 unit 8m 46s hadoop-common in the patch passed.
          +1 unit 0m 52s hadoop-hdfs-client in the patch passed.
          -1 unit 72m 58s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          133m 58s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.namenode.TestReconstructStripedBlocks
            hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:2c91fd8
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806846/h10433_20160528c.patch
          JIRA Issue HDFS-10433
          Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux dd63698998c9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 4e1f56e
          Default Java 1.8.0_91
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15602/artifact/patchprocess/diff-checkstyle-root.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/15602/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15602/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15602/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15602/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 6m 22s trunk passed +1 compile 8m 20s trunk passed +1 checkstyle 1m 25s trunk passed +1 mvnsite 2m 52s trunk passed +1 mvneclipse 0m 40s trunk passed +1 findbugs 5m 36s trunk passed +1 javadoc 2m 18s trunk passed 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 51s the patch passed +1 compile 6m 31s the patch passed +1 javac 6m 31s the patch passed -1 checkstyle 1m 22s root: The patch generated 44 new + 224 unchanged - 6 fixed = 268 total (was 230) +1 mvnsite 2m 52s the patch passed +1 mvneclipse 0m 40s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 5m 38s the patch passed +1 javadoc 2m 48s the patch passed +1 unit 8m 46s hadoop-common in the patch passed. +1 unit 0m 52s hadoop-hdfs-client in the patch passed. -1 unit 72m 58s hadoop-hdfs in the patch failed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 133m 58s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.namenode.TestReconstructStripedBlocks   hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot Subsystem Report/Notes Docker Image:yetus/hadoop:2c91fd8 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806846/h10433_20160528c.patch JIRA Issue HDFS-10433 Optional Tests asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux dd63698998c9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 4e1f56e Default Java 1.8.0_91 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/15602/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/15602/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt unit test logs https://builds.apache.org/job/PreCommit-HDFS-Build/15602/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/15602/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/15602/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          jingzhao Jing Zhao added a comment -

          Thanks for updating the patch, Tsz Wo Nicholas Sze! The latest patch looks good to me overall. Two questions:

          1. The interval between two retries is realized by Thread.sleep, which makes the background thread in AsyncCallHandler sleep. Because all the client side Future.get calls need to wait until the background thread for the final result, this sleep may delay all the pending requests.
          2. The current background thread does a sleep inside of the loop, which may delay all the RPC requests. Ideally we want this thread to wait for response notification from RPC client.
          3. Minor: Counters can be created inside of the Call's constructor method instead of being passed as a parameter.

          Looks like #1 and #2 need some extra work. Considering the current patch is already complicated, we can address them in a separate jira. I will give +1 for committing the current patch first.

          Show
          jingzhao Jing Zhao added a comment - Thanks for updating the patch, Tsz Wo Nicholas Sze ! The latest patch looks good to me overall. Two questions: The interval between two retries is realized by Thread.sleep , which makes the background thread in AsyncCallHandler sleep. Because all the client side Future.get calls need to wait until the background thread for the final result, this sleep may delay all the pending requests. The current background thread does a sleep inside of the loop, which may delay all the RPC requests. Ideally we want this thread to wait for response notification from RPC client. Minor: Counters can be created inside of the Call's constructor method instead of being passed as a parameter. Looks like #1 and #2 need some extra work. Considering the current patch is already complicated, we can address them in a separate jira. I will give +1 for committing the current patch first.
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thanks Jing. I will commit this shortly and file a follow up JIRA.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thanks Jing. I will commit this shortly and file a follow up JIRA.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 5s HADOOP-13226 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806846/h10433_20160528c.patch
          JIRA Issue HADOOP-13226
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9632/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s HADOOP-13226 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12806846/h10433_20160528c.patch JIRA Issue HADOOP-13226 Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/9632/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9889 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9889/)
          HADOOP-13226 Support async call retry and failover. (szetszwo: rev 83f2f78c118a7e52aba5104bd97b0acedc96be7b)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestAsyncIPC.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/AsyncDistributedFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncHDFSWithHA.java
          • hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/AsyncCallHandler.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/CallReturn.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncDFS.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/concurrent/AsyncGet.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9889 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9889/ ) HADOOP-13226 Support async call retry and failover. (szetszwo: rev 83f2f78c118a7e52aba5104bd97b0acedc96be7b) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestAsyncIPC.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/AsyncDistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncHDFSWithHA.java hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/AsyncCallHandler.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/CallReturn.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncDFS.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/concurrent/AsyncGet.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java
          Hide
          szetszwo Tsz Wo Nicholas Sze added a comment -

          Thanks Jing for reviewing the patches and the great ideas!

          I have committed this.

          Show
          szetszwo Tsz Wo Nicholas Sze added a comment - Thanks Jing for reviewing the patches and the great ideas! I have committed this.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #9913 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9913/)
          Revert "HADOOP-13226 Support async call retry and failover." (wang: rev 5360da8bd9f720384860f411bee081aef13b4bd4)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/AsyncCallHandler.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java
          • hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/CallReturn.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestAsyncIPC.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/AsyncDistributedFileSystem.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncDFS.java
          • hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncHDFSWithHA.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/concurrent/AsyncGet.java
          • hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #9913 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9913/ ) Revert " HADOOP-13226 Support async call retry and failover." (wang: rev 5360da8bd9f720384860f411bee081aef13b4bd4) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/AsyncCallHandler.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/CallReturn.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestAsyncIPC.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/AsyncDistributedFileSystem.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncDFS.java hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncHDFSWithHA.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/concurrent/AsyncGet.java hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java

            People

            • Assignee:
              szetszwo Tsz Wo Nicholas Sze
              Reporter:
              xiaobingo Xiaobing Zhou
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development