Details
Description
add a feature that call recoverLease operation automatically when DFSOutputSteam close encounters exception. When we wanted to apply this feature to our cluster, we found that it does not support EC files. HDFS-14694
I think this feature should take effect whether replica files or EC files. This Jira proposes to make it effective when in the case of EC files.
Attachments
Attachments
- HDFS-15684.001.patch
- 2 kB
- Hongbing Wang
- HDFS-15684.002.patch
- 7 kB
- Hongbing Wang
- HDFS-15684.003.patch
- 7 kB
- Hongbing Wang
Issue Links
- is part of
-
HDFS-14694 Call recoverLease on DFSOutputStream close exception
- Resolved
- relates to
-
HDFS-15858 Backport HDFS-14694 to branch-3.1/3.2/3.3
- Patch Available
Activity
-1 overall |
Vote | Subsystem | Runtime | Logfile | Comment |
---|---|---|---|---|
0 | reexec | 37m 6s | Docker mode activated. | |
Prechecks | ||||
+1 | dupname | 0m 0s | No case conflicting files found. | |
+1 | @author | 0m 0s | The patch does not contain any @author tags. | |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
trunk Compile Tests | ||||
0 | mvndep | 2m 7s | Maven dependency ordering for branch | |
+1 | mvninstall | 28m 3s | trunk passed | |
+1 | compile | 5m 20s | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 | |
+1 | compile | 5m 2s | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | |
+1 | checkstyle | 1m 11s | trunk passed | |
+1 | mvnsite | 2m 35s | trunk passed | |
+1 | shadedclient | 20m 33s | branch has no errors when building and testing our client artifacts. | |
+1 | javadoc | 1m 27s | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 | |
+1 | javadoc | 1m 55s | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | |
0 | spotbugs | 3m 19s | Used deprecated FindBugs config; considering switching to SpotBugs. | |
+1 | findbugs | 5m 49s | trunk passed | |
Patch Compile Tests | ||||
0 | mvndep | 0m 23s | Maven dependency ordering for patch | |
+1 | mvninstall | 2m 15s | the patch passed | |
+1 | compile | 5m 3s | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 | |
+1 | javac | 5m 3s | the patch passed | |
+1 | compile | 5m 4s | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | |
+1 | javac | 5m 4s | the patch passed | |
+1 | blanks | 0m 0s | The patch has no blanks issues. | |
-0 | checkstyle | 0m 52s | /results-checkstyle-hadoop-hdfs-project.txt | hadoop-hdfs-project: The patch generated 4 new + 33 unchanged - 0 fixed = 37 total (was 33) |
+1 | mvnsite | 2m 0s | the patch passed | |
+1 | shadedclient | 20m 56s | patch has no errors when building and testing our client artifacts. | |
+1 | javadoc | 1m 25s | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 | |
+1 | javadoc | 1m 49s | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 | |
+1 | findbugs | 5m 53s | the patch passed | |
Other Tests | ||||
+1 | unit | 2m 18s | hadoop-hdfs-client in the patch passed. | |
-1 | unit | 131m 40s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
+1 | asflicense | 0m 41s | The patch does not generate ASF License warnings. | |
286m 50s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.TestDFSInotifyEventInputStreamKerberized |
hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints | |
hadoop.hdfs.server.namenode.TestFileTruncate | |
hadoop.hdfs.TestCrcCorruption |
Subsystem | Report/Notes |
---|---|
Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/304/artifact/out/Dockerfile |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13015360/HDFS-15684.002.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
uname | Linux e9b7ab1ca3dc 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | personality/hadoop.sh |
git revision | trunk / dd85a90da6f3c475915def97ace73bb5162e3033 |
Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 |
Test Results | https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/304/testReport/ |
Max. process+thread count | 2971 (vs. ulimit of 5500) |
modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
Console output | https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/304/console |
versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
wanghongbing Thanks for report and fix. It looks good. Could you please fix the checkstyle?
hexiaoqiao leosun08 It is similar to HDFS-14694, Could you please take another look?
Thanks wanghongbing and ferhui for your works. +1 from my side once fix the checkstyle.
-1 overall |
Vote | Subsystem | Runtime | Logfile | Comment |
---|---|---|---|---|
0 | reexec | 0m 40s | Docker mode activated. | |
Prechecks | ||||
+1 | dupname | 0m 0s | No case conflicting files found. | |
0 | codespell | 0m 1s | codespell was not available. | |
+1 | @author | 0m 0s | The patch does not contain any @author tags. | |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
trunk Compile Tests | ||||
0 | mvndep | 1m 56s | Maven dependency ordering for branch | |
+1 | mvninstall | 20m 42s | trunk passed | |
+1 | compile | 4m 19s | trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 | |
+1 | compile | 3m 47s | trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 | |
+1 | checkstyle | 0m 56s | trunk passed | |
+1 | mvnsite | 2m 17s | trunk passed | |
+1 | shadedclient | 18m 32s | branch has no errors when building and testing our client artifacts. | |
+1 | javadoc | 1m 34s | trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 | |
+1 | javadoc | 2m 3s | trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 | |
0 | spotbugs | 3m 9s | Used deprecated FindBugs config; considering switching to SpotBugs. | |
+1 | findbugs | 5m 33s | trunk passed | |
Patch Compile Tests | ||||
0 | mvndep | 0m 27s | Maven dependency ordering for patch | |
+1 | mvninstall | 2m 1s | the patch passed | |
+1 | compile | 4m 2s | the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 | |
+1 | javac | 4m 2s | the patch passed | |
+1 | compile | 3m 41s | the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 | |
+1 | javac | 3m 41s | the patch passed | |
+1 | blanks | 0m 0s | The patch has no blanks issues. | |
+1 | checkstyle | 0m 50s | the patch passed | |
+1 | mvnsite | 2m 2s | the patch passed | |
+1 | shadedclient | 18m 43s | patch has no errors when building and testing our client artifacts. | |
+1 | javadoc | 1m 25s | the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 | |
+1 | javadoc | 1m 52s | the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 | |
+1 | findbugs | 5m 41s | the patch passed | |
Other Tests | ||||
+1 | unit | 2m 19s | hadoop-hdfs-client in the patch passed. | |
-1 | unit | 114m 38s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
-1 | asflicense | 0m 46s | /results-asflicense.txt | The patch generated 2 ASF License warnings. |
217m 7s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.TestDisableConnCache |
hadoop.hdfs.server.namenode.TestNamenodeRetryCache | |
hadoop.hdfs.TestDatanodeReport | |
hadoop.hdfs.TestPread | |
hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy | |
hadoop.hdfs.TestDecommission | |
hadoop.hdfs.TestGetFileChecksum | |
hadoop.hdfs.server.namenode.TestAddStripedBlockInFBR | |
hadoop.hdfs.TestDecommissionWithStripedBackoffMonitor | |
hadoop.hdfs.TestParallelUnixDomainRead | |
hadoop.hdfs.server.namenode.sps.TestStoragePolicySatisfierWithStripedFile | |
hadoop.hdfs.TestErasureCodingPolicies | |
hadoop.hdfs.client.impl.TestBlockReaderLocalLegacy | |
hadoop.hdfs.TestDistributedFileSystemWithECFileWithRandomECPolicy | |
hadoop.hdfs.TestMiniDFSCluster | |
hadoop.hdfs.TestReconstructStripedFile | |
hadoop.hdfs.TestMaintenanceState | |
hadoop.hdfs.TestFileAppend2 | |
hadoop.hdfs.TestDFSStripedOutputStream | |
hadoop.hdfs.TestReadStripedFileWithDNFailure | |
hadoop.hdfs.server.balancer.TestBalancer | |
hadoop.hdfs.TestDFSPermission | |
hadoop.hdfs.server.namenode.TestFSImage | |
hadoop.hdfs.TestDFSOutputStream | |
hadoop.hdfs.TestDFSClientSocketSize | |
hadoop.hdfs.TestDFSStripedOutputStreamWithRandomECPolicy | |
hadoop.hdfs.client.impl.TestBlockReaderLocal | |
hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy | |
hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped | |
hadoop.hdfs.TestEncryptionZones | |
hadoop.hdfs.TestReadStripedFileWithMissingBlocks | |
hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy |
Subsystem | Report/Notes |
---|---|
Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/307/artifact/out/Dockerfile |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13015695/HDFS-15684.003.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle codespell |
uname | Linux e1787e5d1507 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | personality/hadoop.sh |
git revision | trunk / 8fa699b53fea8728e008c46af949f92543c08170 |
Default Java | Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
Test Results | https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/307/testReport/ |
Max. process+thread count | 3266 (vs. ulimit of 5500) |
modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
Console output | https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/307/console |
versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
`TestDFSOutputStream` pass in local. Other OOM failed tests also pass in local when Randomly testing some.
-1 overall |
Vote | Subsystem | Runtime | Logfile | Comment |
---|---|---|---|---|
0 | reexec | 0m 48s | Docker mode activated. | |
Prechecks | ||||
+1 | dupname | 0m 0s | No case conflicting files found. | |
0 | codespell | 0m 0s | codespell was not available. | |
+1 | @author | 0m 0s | The patch does not contain any @author tags. | |
+1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
trunk Compile Tests | ||||
0 | mvndep | 1m 55s | Maven dependency ordering for branch | |
+1 | mvninstall | 23m 3s | trunk passed | |
+1 | compile | 4m 22s | trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 | |
+1 | compile | 3m 54s | trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 | |
+1 | checkstyle | 0m 56s | trunk passed | |
+1 | mvnsite | 2m 8s | trunk passed | |
+1 | shadedclient | 19m 49s | branch has no errors when building and testing our client artifacts. | |
+1 | javadoc | 1m 27s | trunk passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 | |
+1 | javadoc | 1m 55s | trunk passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 | |
0 | spotbugs | 3m 10s | Used deprecated FindBugs config; considering switching to SpotBugs. | |
+1 | findbugs | 5m 40s | trunk passed | |
Patch Compile Tests | ||||
0 | mvndep | 0m 22s | Maven dependency ordering for patch | |
+1 | mvninstall | 1m 58s | the patch passed | |
+1 | compile | 4m 13s | the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 | |
+1 | javac | 4m 13s | the patch passed | |
+1 | compile | 3m 47s | the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 | |
+1 | javac | 3m 47s | the patch passed | |
+1 | blanks | 0m 1s | The patch has no blanks issues. | |
+1 | checkstyle | 0m 52s | the patch passed | |
+1 | mvnsite | 2m 0s | the patch passed | |
+1 | shadedclient | 20m 40s | patch has no errors when building and testing our client artifacts. | |
+1 | javadoc | 1m 28s | the patch passed with JDK Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 | |
+1 | javadoc | 1m 51s | the patch passed with JDK Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 | |
+1 | findbugs | 6m 12s | the patch passed | |
Other Tests | ||||
+1 | unit | 2m 15s | hadoop-hdfs-client in the patch passed. | |
-1 | unit | 129m 49s | /patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt | hadoop-hdfs in the patch passed. |
+1 | asflicense | 0m 53s | The patch does not generate ASF License warnings. | |
238m 23s |
Reason | Tests |
---|---|
Failed junit tests | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
hadoop.hdfs.TestMultipleNNPortQOP | |
hadoop.hdfs.server.namenode.TestFSImage |
Subsystem | Report/Notes |
---|---|
Docker | ClientAPI=1.40 ServerAPI=1.40 base: https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/308/artifact/out/Dockerfile |
JIRA Issue | |
JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/13015695/HDFS-15684.003.patch |
Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle codespell |
uname | Linux ed66460ced89 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9 23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
Build tool | maven |
Personality | personality/hadoop.sh |
git revision | trunk / 740399a602d7ad0cd63eaccd2de6d5a506f3e913 |
Default Java | Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 |
Test Results | https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/308/testReport/ |
Max. process+thread count | 2704 (vs. ulimit of 5500) |
modules | C: hadoop-hdfs-project/hadoop-hdfs-client hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project |
Console output | https://ci-hadoop.apache.org/job/PreCommit-HDFS-Build/308/console |
versions | git=2.17.1 maven=3.6.0 findbugs=4.0.6 |
Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
+1 for HDFS-15684.003.patch. Will commit shortly once ferhui give another confirms.
Committed to trunk.
Thanks wanghongbing for contribution! Thanks ferhui for your reviews!
wanghongbing cherry-pick failed, could you please give patches on branch 3.3, 3.2, 3.1? Thanks
ferhui ok. Because this PR depends on I will backport them in another Jira later.HDFS-14694,
HDFS-15684This message was automatically generated.