Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
-
Reviewed
Description
In current Async DFS implementation, file system calls are invoked and returns Future<?> immediately to clients. Clients call Future#get to retrieve final results. Future#get internally invokes a chain of callbacks residing in ClientNamenodeProtocolTranslatorPB, ProtobufRpcEngine and ipc.Client. The callback path bypasses the original retry layer/logic designed for synchronous DFS. This proposes refactoring to make retry also works for Async DFS.
Attachments
Attachments
- h10433_20160528c.patch
- 47 kB
- Tsz-wo Sze
- h10433_20160528.patch
- 45 kB
- Tsz-wo Sze
- h10433_20160527.patch
- 49 kB
- Tsz-wo Sze
- h10433_20160525b.patch
- 37 kB
- Tsz-wo Sze
- h10433_20160525.patch
- 37 kB
- Tsz-wo Sze
- h10433_20160524.patch
- 31 kB
- Tsz-wo Sze
Issue Links
- relates to
-
HADOOP-13227 AsyncCallHandler should use an event driven architecture to handle async calls
- Resolved
Activity
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 15s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 2 new or modified test files. |
0 | mvndep | 0m 12s | Maven dependency ordering for branch |
+1 | mvninstall | 6m 45s | trunk passed |
+1 | compile | 7m 13s | trunk passed |
+1 | checkstyle | 1m 37s | trunk passed |
+1 | mvnsite | 2m 42s | trunk passed |
+1 | mvneclipse | 0m 41s | trunk passed |
+1 | findbugs | 4m 50s | trunk passed |
+1 | javadoc | 2m 32s | trunk passed |
0 | mvndep | 0m 12s | Maven dependency ordering for patch |
+1 | mvninstall | 2m 16s | the patch passed |
+1 | compile | 7m 26s | the patch passed |
-1 | javac | 9m 54s | root generated 1 new + 695 unchanged - 1 fixed = 696 total (was 696) |
+1 | javac | 7m 26s | the patch passed |
-1 | checkstyle | 1m 34s | root: patch generated 42 new + 211 unchanged - 5 fixed = 253 total (was 216) |
+1 | mvnsite | 2m 50s | the patch passed |
+1 | mvneclipse | 0m 43s | the patch passed |
+1 | whitespace | 0m 0s | Patch has no whitespace issues. |
-1 | findbugs | 1m 44s | hadoop-common-project/hadoop-common generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) |
+1 | javadoc | 2m 40s | the patch passed |
+1 | unit | 9m 9s | hadoop-common in the patch passed. |
+1 | unit | 0m 57s | hadoop-hdfs-client in the patch passed. |
-1 | unit | 92m 6s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 35s | Patch does not generate ASF License warnings. |
153m 59s |
Reason | Tests |
---|---|
FindBugs | module:hadoop-common-project/hadoop-common |
Inconsistent synchronization of org.apache.hadoop.io.retry.RetryInvocationHandler$CallReturn.returnValue; locked 66% of time Unsynchronized access at RetryInvocationHandler.java:66% of time Unsynchronized access at RetryInvocationHandler.java:[line 219] | |
Inconsistent synchronization of org.apache.hadoop.io.retry.RetryInvocationHandler$CallReturn.state; locked 83% of time Unsynchronized access at RetryInvocationHandler.java:83% of time Unsynchronized access at RetryInvocationHandler.java:[line 219] | |
Inconsistent synchronization of org.apache.hadoop.io.retry.RetryInvocationHandler$CallReturn.thrown; locked 80% of time Unsynchronized access at RetryInvocationHandler.java:80% of time Unsynchronized access at RetryInvocationHandler.java:[line 219] | |
Wait not in loop in org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java:org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java:[line 56] | |
Failed junit tests | hadoop.hdfs.TestAsyncDFS |
hadoop.hdfs.server.datanode.TestFsDatasetCache | |
hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations | |
hadoop.hdfs.server.namenode.TestEditLog | |
hadoop.hdfs.server.namenode.TestFileTruncate | |
hadoop.hdfs.TestAsyncDFSRename | |
Timed out junit tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 20s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
0 | mvndep | 0m 37s | Maven dependency ordering for branch |
+1 | mvninstall | 6m 42s | trunk passed |
+1 | compile | 6m 33s | trunk passed |
+1 | checkstyle | 1m 26s | trunk passed |
+1 | mvnsite | 2m 42s | trunk passed |
+1 | mvneclipse | 0m 58s | trunk passed |
+1 | findbugs | 4m 19s | trunk passed |
+1 | javadoc | 2m 28s | trunk passed |
0 | mvndep | 0m 11s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 58s | the patch passed |
+1 | compile | 7m 18s | the patch passed |
+1 | javac | 7m 18s | the patch passed |
-1 | checkstyle | 1m 30s | root: patch generated 30 new + 225 unchanged - 6 fixed = 255 total (was 231) |
+1 | mvnsite | 2m 39s | the patch passed |
+1 | mvneclipse | 0m 34s | the patch passed |
+1 | whitespace | 0m 0s | Patch has no whitespace issues. |
+1 | xml | 0m 2s | The patch has no ill-formed XML file. |
-1 | findbugs | 1m 40s | hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
+1 | javadoc | 2m 27s | the patch passed |
+1 | unit | 8m 52s | hadoop-common in the patch passed. |
+1 | unit | 0m 58s | hadoop-hdfs-client in the patch passed. |
-1 | unit | 66m 16s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 26s | Patch does not generate ASF License warnings. |
125m 44s |
Reason | Tests |
---|---|
FindBugs | module:hadoop-common-project/hadoop-common |
Wait not in loop in org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java:org.apache.hadoop.util.concurrent.AsyncGet$Util.wait(Object, long, TimeUnit) At AsyncGet.java:[line 56] | |
Failed junit tests | hadoop.hdfs.TestAsyncDFSRename |
hadoop.hdfs.TestErasureCodeBenchmarkThroughput |
This message was automatically generated.
h10433_20160525b.patch: fixes more warnings.
Note that the indentation checkstyle warnings are bogus.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 11s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
0 | mvndep | 0m 11s | Maven dependency ordering for branch |
+1 | mvninstall | 6m 21s | trunk passed |
+1 | compile | 6m 25s | trunk passed |
+1 | checkstyle | 1m 22s | trunk passed |
+1 | mvnsite | 2m 15s | trunk passed |
+1 | mvneclipse | 0m 35s | trunk passed |
+1 | findbugs | 4m 22s | trunk passed |
+1 | javadoc | 2m 21s | trunk passed |
0 | mvndep | 0m 11s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 58s | the patch passed |
+1 | compile | 6m 15s | the patch passed |
+1 | javac | 6m 15s | the patch passed |
-1 | checkstyle | 1m 22s | root: patch generated 26 new + 224 unchanged - 6 fixed = 250 total (was 230) |
+1 | mvnsite | 2m 17s | the patch passed |
+1 | mvneclipse | 0m 35s | the patch passed |
+1 | whitespace | 0m 0s | Patch has no whitespace issues. |
+1 | xml | 0m 2s | The patch has no ill-formed XML file. |
+1 | findbugs | 4m 55s | the patch passed |
+1 | javadoc | 2m 20s | the patch passed |
+1 | unit | 7m 7s | hadoop-common in the patch passed. |
+1 | unit | 0m 54s | hadoop-hdfs-client in the patch passed. |
+1 | unit | 58m 2s | hadoop-hdfs in the patch passed. |
+1 | asflicense | 0m 24s | Patch does not generate ASF License warnings. |
111m 19s |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:2c91fd8 |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12806273/h10433_20160525b.patch |
JIRA Issue | |
Optional Tests | asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux 595da50cdba9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 013532a |
Default Java | 1.8.0_91 |
findbugs | v3.0.0 |
checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/15572/artifact/patchprocess/diff-checkstyle-root.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/15572/testReport/ |
modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/15572/console |
Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
Thanks Jing for the offline discussion. Here is a new patch.
h10433_20160527.patch:
- refactors async logic to AsyncCallHandler,
- fixes a bug for the proxies without retry, and
- add a new test, testAsyncWithoutRetry.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 0s | Docker mode activated. |
-1 | patch | 0m 5s | |
Subsystem | Report/Notes |
---|---|
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12806785/h10433_20160527.patch |
JIRA Issue | |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/15597/console |
Powered by | Apache Yetus 0.2.0 http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 17s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
0 | mvndep | 0m 39s | Maven dependency ordering for branch |
+1 | mvninstall | 6m 10s | trunk passed |
+1 | compile | 6m 13s | trunk passed |
+1 | checkstyle | 1m 22s | trunk passed |
+1 | mvnsite | 2m 15s | trunk passed |
+1 | mvneclipse | 0m 35s | trunk passed |
+1 | findbugs | 4m 17s | trunk passed |
+1 | javadoc | 2m 16s | trunk passed |
0 | mvndep | 0m 12s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 52s | the patch passed |
+1 | compile | 6m 11s | the patch passed |
+1 | javac | 6m 11s | the patch passed |
-1 | checkstyle | 1m 20s | root: The patch generated 46 new + 224 unchanged - 6 fixed = 270 total (was 230) |
+1 | mvnsite | 2m 11s | the patch passed |
+1 | mvneclipse | 0m 34s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. |
+1 | findbugs | 4m 40s | the patch passed |
+1 | javadoc | 2m 17s | the patch passed |
+1 | unit | 12m 49s | hadoop-common in the patch passed. |
+1 | unit | 0m 54s | hadoop-hdfs-client in the patch passed. |
-1 | unit | 74m 14s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 27s | The patch does not generate ASF License warnings. |
132m 44s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.TestErasureCodeBenchmarkThroughput |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:2c91fd8 |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12806827/h10433_20160528.patch |
JIRA Issue | |
Optional Tests | asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux ebb0db7a1a3a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / f5ff05c |
Default Java | 1.8.0_91 |
findbugs | v3.0.0 |
checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/15598/artifact/patchprocess/diff-checkstyle-root.txt |
unit | https://builds.apache.org/job/PreCommit-HDFS-Build/15598/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
unit test logs | https://builds.apache.org/job/PreCommit-HDFS-Build/15598/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/15598/testReport/ |
modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/15598/console |
Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
h10433_20160528b.patch: makes testAsyncWithoutRetry to be run in both HA and non-HA cases.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 15s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
0 | mvndep | 0m 48s | Maven dependency ordering for branch |
+1 | mvninstall | 6m 15s | trunk passed |
+1 | compile | 6m 14s | trunk passed |
+1 | checkstyle | 1m 24s | trunk passed |
+1 | mvnsite | 2m 30s | trunk passed |
+1 | mvneclipse | 0m 35s | trunk passed |
+1 | findbugs | 5m 16s | trunk passed |
+1 | javadoc | 2m 29s | trunk passed |
0 | mvndep | 0m 15s | Maven dependency ordering for patch |
+1 | mvninstall | 2m 29s | the patch passed |
+1 | compile | 8m 15s | the patch passed |
+1 | javac | 8m 15s | the patch passed |
-1 | checkstyle | 1m 22s | root: The patch generated 44 new + 224 unchanged - 6 fixed = 268 total (was 230) |
+1 | mvnsite | 2m 12s | the patch passed |
+1 | mvneclipse | 0m 35s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. |
+1 | findbugs | 4m 50s | the patch passed |
+1 | javadoc | 2m 21s | the patch passed |
+1 | unit | 9m 28s | hadoop-common in the patch passed. |
+1 | unit | 1m 8s | hadoop-hdfs-client in the patch passed. |
-1 | unit | 76m 30s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 22s | The patch does not generate ASF License warnings. |
136m 37s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.TestPersistBlocks |
hadoop.hdfs.TestAsyncHDFSWithHA |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:2c91fd8 |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12806837/h10433_20160528b.patch |
JIRA Issue | |
Optional Tests | asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux 990848a97abf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / f5ff05c |
Default Java | 1.8.0_91 |
findbugs | v3.0.0 |
checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/15599/artifact/patchprocess/diff-checkstyle-root.txt |
unit | https://builds.apache.org/job/PreCommit-HDFS-Build/15599/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
unit test logs | https://builds.apache.org/job/PreCommit-HDFS-Build/15599/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/15599/testReport/ |
modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/15599/console |
Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 14s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
0 | mvndep | 0m 17s | Maven dependency ordering for branch |
+1 | mvninstall | 6m 25s | trunk passed |
+1 | compile | 8m 22s | trunk passed |
+1 | checkstyle | 1m 23s | trunk passed |
+1 | mvnsite | 2m 55s | trunk passed |
+1 | mvneclipse | 0m 39s | trunk passed |
+1 | findbugs | 5m 32s | trunk passed |
+1 | javadoc | 2m 19s | trunk passed |
0 | mvndep | 0m 11s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 53s | the patch passed |
+1 | compile | 6m 36s | the patch passed |
+1 | javac | 6m 36s | the patch passed |
-1 | checkstyle | 1m 22s | root: The patch generated 44 new + 224 unchanged - 6 fixed = 268 total (was 230) |
+1 | mvnsite | 2m 51s | the patch passed |
+1 | mvneclipse | 0m 42s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 1s | The patch has no ill-formed XML file. |
+1 | findbugs | 5m 37s | the patch passed |
+1 | javadoc | 2m 44s | the patch passed |
+1 | unit | 8m 41s | hadoop-common in the patch passed. |
+1 | unit | 0m 57s | hadoop-hdfs-client in the patch passed. |
-1 | unit | 75m 47s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 22s | The patch does not generate ASF License warnings. |
136m 50s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
hadoop.hdfs.TestAsyncHDFSWithHA | |
hadoop.hdfs.server.datanode.TestDataNodeErasureCodingMetrics |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:2c91fd8 |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12806837/h10433_20160528b.patch |
JIRA Issue | |
Optional Tests | asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux b56bdee95ca5 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / f5ff05c |
Default Java | 1.8.0_91 |
findbugs | v3.0.0 |
checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/15600/artifact/patchprocess/diff-checkstyle-root.txt |
unit | https://builds.apache.org/job/PreCommit-HDFS-Build/15600/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
unit test logs | https://builds.apache.org/job/PreCommit-HDFS-Build/15600/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/15600/testReport/ |
modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/15600/console |
Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
Jenkins somehow tested h10433_20160528b.patch again but not /h10433_20160528c.patch.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 16s | Docker mode activated. |
+1 | @author | 0m 0s | The patch does not contain any @author tags. |
+1 | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
0 | mvndep | 0m 11s | Maven dependency ordering for branch |
+1 | mvninstall | 6m 22s | trunk passed |
+1 | compile | 8m 20s | trunk passed |
+1 | checkstyle | 1m 25s | trunk passed |
+1 | mvnsite | 2m 52s | trunk passed |
+1 | mvneclipse | 0m 40s | trunk passed |
+1 | findbugs | 5m 36s | trunk passed |
+1 | javadoc | 2m 18s | trunk passed |
0 | mvndep | 0m 12s | Maven dependency ordering for patch |
+1 | mvninstall | 1m 51s | the patch passed |
+1 | compile | 6m 31s | the patch passed |
+1 | javac | 6m 31s | the patch passed |
-1 | checkstyle | 1m 22s | root: The patch generated 44 new + 224 unchanged - 6 fixed = 268 total (was 230) |
+1 | mvnsite | 2m 52s | the patch passed |
+1 | mvneclipse | 0m 40s | the patch passed |
+1 | whitespace | 0m 0s | The patch has no whitespace issues. |
+1 | xml | 0m 2s | The patch has no ill-formed XML file. |
+1 | findbugs | 5m 38s | the patch passed |
+1 | javadoc | 2m 48s | the patch passed |
+1 | unit | 8m 46s | hadoop-common in the patch passed. |
+1 | unit | 0m 52s | hadoop-hdfs-client in the patch passed. |
-1 | unit | 72m 58s | hadoop-hdfs in the patch failed. |
+1 | asflicense | 0m 24s | The patch does not generate ASF License warnings. |
133m 58s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
hadoop.hdfs.server.namenode.TestReconstructStripedBlocks | |
hadoop.hdfs.server.namenode.snapshot.TestOpenFilesWithSnapshot |
Subsystem | Report/Notes |
---|---|
Docker | Image:yetus/hadoop:2c91fd8 |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12806846/h10433_20160528c.patch |
JIRA Issue | |
Optional Tests | asflicense xml compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
uname | Linux dd63698998c9 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
git revision | trunk / 4e1f56e |
Default Java | 1.8.0_91 |
findbugs | v3.0.0 |
checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/15602/artifact/patchprocess/diff-checkstyle-root.txt |
unit | https://builds.apache.org/job/PreCommit-HDFS-Build/15602/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
unit test logs | https://builds.apache.org/job/PreCommit-HDFS-Build/15602/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/15602/testReport/ |
modules | C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: . |
Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/15602/console |
Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
Thanks for updating the patch, szetszwo! The latest patch looks good to me overall. Two questions:
- The interval between two retries is realized by Thread.sleep, which makes the background thread in AsyncCallHandler sleep. Because all the client side Future.get calls need to wait until the background thread for the final result, this sleep may delay all the pending requests.
- The current background thread does a sleep inside of the loop, which may delay all the RPC requests. Ideally we want this thread to wait for response notification from RPC client.
- Minor: Counters can be created inside of the Call's constructor method instead of being passed as a parameter.
Looks like #1 and #2 need some extra work. Considering the current patch is already complicated, we can address them in a separate jira. I will give +1 for committing the current patch first.
-1 overall |
Vote | Subsystem | Runtime | Comment |
---|---|---|---|
0 | reexec | 0m 0s | Docker mode activated. |
-1 | patch | 0m 5s | |
Subsystem | Report/Notes |
---|---|
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12806846/h10433_20160528c.patch |
JIRA Issue | |
Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/9632/console |
Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
SUCCESS: Integrated in Hadoop-trunk-Commit #9889 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9889/)
HADOOP-13226 Support async call retry and failover. (szetszwo: rev 83f2f78c118a7e52aba5104bd97b0acedc96be7b)
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
- hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestAsyncIPC.java
- hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/AsyncDistributedFileSystem.java
- hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncHDFSWithHA.java
- hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/AsyncCallHandler.java
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/CallReturn.java
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncDFS.java
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/concurrent/AsyncGet.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java
Thanks Jing for reviewing the patches and the great ideas!
I have committed this.
SUCCESS: Integrated in Hadoop-trunk-Commit #9913 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9913/)
Revert "HADOOP-13226 Support async call retry and failover." (wang: rev 5360da8bd9f720384860f411bee081aef13b4bd4)
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/AsyncCallHandler.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/HATestUtil.java
- hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/CallReturn.java
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/ProtobufRpcEngine.java
- hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ipc/TestAsyncIPC.java
- hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/AsyncDistributedFileSystem.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncDFS.java
- hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestAsyncHDFSWithHA.java
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryInvocationHandler.java
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/concurrent/AsyncGet.java
- hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientNamenodeProtocolTranslatorPB.java
- hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java
h10433_20160524.patch: adds an async call queue to RetryInvocationHandler to support retry and failover.