Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Critical
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 2.9.0, 2.7.4, 3.0.0-alpha4, 2.8.2
-
Component/s: None
-
Labels:None
-
Hadoop Flags:Reviewed
Description
In the following scenario,FSCK shows overall health status as corrupt even it's has one good replica.
1. Create file with 2 RF.
2. Shutdown one DN
3. Append to file again.
4. Restart the DN
5. After block report, check Fsck
Issue Links
- is duplicated by
-
HDFS-11797
BlockManager#createLocatedBlocks() can throw ArrayIndexOutofBoundsException when corrupt replicas are inconsistent
-
- Resolved
-
-
HDFS-12630
Rolling restart can create inconsistency between blockMap and corrupt replicas map
-
- Resolved
-
-
HDFS-12676
when blocks has corrupted replicas，throws Exception
-
- Resolved
-
- relates to
-
HDFS-11755
Underconstruction blocks can be considered missing
-
- Resolved
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 23s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 13m 38s | trunk passed |
| +1 | compile | 0m 46s | trunk passed |
| +1 | checkstyle | 0m 35s | trunk passed |
| +1 | mvnsite | 0m 52s | trunk passed |
| +1 | mvneclipse | 0m 13s | trunk passed |
| +1 | findbugs | 1m 46s | trunk passed |
| +1 | javadoc | 0m 40s | trunk passed |
| +1 | mvninstall | 0m 47s | the patch passed |
| +1 | compile | 0m 45s | the patch passed |
| +1 | javac | 0m 45s | the patch passed |
| +1 | checkstyle | 0m 33s | the patch passed |
| +1 | mvnsite | 0m 48s | the patch passed |
| +1 | mvneclipse | 0m 10s | the patch passed |
| +1 | whitespace | 0m 1s | The patch has no whitespace issues. |
| +1 | findbugs | 1m 51s | the patch passed |
| +1 | javadoc | 0m 39s | the patch passed |
| -1 | unit | 74m 31s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 29s | The patch does not generate ASF License warnings. |
| 100m 47s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.TestMaintenanceState |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureToleration | |
| hadoop.hdfs.server.datanode.TestDirectoryScanner | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12854146/HDFS-11445.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 762077f73dda 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 13d4bcf |
| Default Java | 1.8.0_121 |
| findbugs | v3.0.0 |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/18423/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/18423/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/18423/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Test failures are unrelated..Kindly review.
Hi Jing Zhao/Kihwal Lee/Tsz Wo Nicholas Sze
If you get chance, can you please review..?
Thanks for working on this, Brahma Reddy Battula!
I think you just find a scenario that the following inconsistency happens:
BlockManager#createLocatedBlock
NumberReplicas numReplicas = countNodes(blk);
final int numCorruptNodes = numReplicas.corruptReplicas();
final int numCorruptReplicas = corruptReplicas.numCorruptReplicas(blk);
if (numCorruptNodes != numCorruptReplicas) {
LOG.warn("Inconsistent number of corrupt replicas for "
+ blk + " blockMap has " + numCorruptNodes
+ " but corrupt replicas map has " + numCorruptReplicas);
}
I also did some debugging using your unit test. Looks like the root cause for this inconsistency is: BlockInfo#setGenerationStampAndVerifyReplicas may remove a datanode storage from the block's storage list, but still leave the storage in the CorruptReplicasMap.
This inconsistency later can be fixed automatically, e.g., by a full block report. But maybe we should consider using BlockManager#removeStoredBlock(BlockInfo, DatanodeDescriptor) to remove all the records related to the block-dn pair.
Jing Zhao thanks a lot for your review..
This inconsistency later can be fixed automatically, e.g., by a full block report. But maybe we should consider using BlockManager#removeStoredBlock(BlockInfo, DatanodeDescriptor) to remove all the records related to the block-dn pair
Yes,Updated the patch. Kindly review.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 15s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 15m 10s | trunk passed |
| +1 | compile | 0m 48s | trunk passed |
| +1 | checkstyle | 0m 40s | trunk passed |
| +1 | mvnsite | 0m 57s | trunk passed |
| +1 | mvneclipse | 0m 15s | trunk passed |
| -1 | findbugs | 2m 2s | hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. |
| +1 | javadoc | 0m 47s | trunk passed |
| +1 | mvninstall | 0m 57s | the patch passed |
| +1 | compile | 0m 54s | the patch passed |
| +1 | javac | 0m 54s | the patch passed |
| -0 | checkstyle | 0m 39s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 5 new + 310 unchanged - 1 fixed = 315 total (was 311) |
| +1 | mvnsite | 1m 0s | the patch passed |
| +1 | mvneclipse | 0m 13s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 2m 0s | the patch passed |
| +1 | javadoc | 0m 43s | the patch passed |
| -1 | unit | 73m 46s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 19s | The patch does not generate ASF License warnings. |
| 102m 56s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
This message was automatically generated.
Thanks for the patch, Brahma Reddy Battula. For the current patch, instead of passing BlockManager to commitBlock and setGenerationStampAnderifyReplcias, why not letting these methods return the list of stale replicas and then removing these stored blockInfo in BlockManager?
why not letting these methods return the list of stale replicas and then removing these stored blockInfo in BlockManager?
Make sense, Updated the patch, kindly review.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 25s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 14m 4s | trunk passed |
| +1 | compile | 0m 49s | trunk passed |
| +1 | checkstyle | 0m 40s | trunk passed |
| +1 | mvnsite | 0m 53s | trunk passed |
| +1 | mvneclipse | 0m 15s | trunk passed |
| +1 | findbugs | 1m 42s | trunk passed |
| +1 | javadoc | 0m 41s | trunk passed |
| +1 | mvninstall | 0m 50s | the patch passed |
| +1 | compile | 0m 45s | the patch passed |
| +1 | javac | 0m 45s | the patch passed |
| -0 | checkstyle | 0m 36s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 309 unchanged - 2 fixed = 312 total (was 311) |
| +1 | mvnsite | 0m 52s | the patch passed |
| +1 | mvneclipse | 0m 12s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 1m 46s | the patch passed |
| +1 | javadoc | 0m 39s | the patch passed |
| -1 | unit | 68m 59s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 21s | The patch does not generate ASF License warnings. |
| 95m 55s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 | |
| hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure | |
| Timed out junit tests | org.apache.hadoop.hdfs.server.blockmanagement.TestBlockStatsMXBean |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12868531/HDFS-11445-003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 5ab1f9862b5c 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 035d468 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/19469/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/19469/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/19469/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/19469/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Thanks for updating the patch, Brahma Reddy Battula. The patch looks good to me. +1 after fixing the new checkstyle warnings.
Jing Zhao thanks for review..
Updated the patch to Address the check-style.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 18s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 15m 43s | trunk passed |
| +1 | compile | 0m 47s | trunk passed |
| +1 | checkstyle | 0m 40s | trunk passed |
| +1 | mvnsite | 0m 52s | trunk passed |
| +1 | mvneclipse | 0m 15s | trunk passed |
| +1 | findbugs | 1m 40s | trunk passed |
| +1 | javadoc | 0m 38s | trunk passed |
| +1 | mvninstall | 0m 48s | the patch passed |
| +1 | compile | 0m 44s | the patch passed |
| +1 | javac | 0m 44s | the patch passed |
| -0 | checkstyle | 0m 36s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 307 unchanged - 2 fixed = 308 total (was 309) |
| +1 | mvnsite | 0m 49s | the patch passed |
| +1 | mvneclipse | 0m 12s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 1m 45s | the patch passed |
| +1 | javadoc | 0m 36s | the patch passed |
| -1 | unit | 63m 3s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 19s | The patch does not generate ASF License warnings. |
| 91m 1s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| hadoop.hdfs.server.namenode.ha.TestFailureToReadEdits |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12869154/HDFS-11445-004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 69ae63eaa026 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / e135c0b |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/19527/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/19527/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/19527/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/19527/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Uploaded the branch-2 which are having minimal conflicts,As EC is not present.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 15s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 15m 9s | trunk passed |
| +1 | compile | 1m 0s | trunk passed |
| +1 | checkstyle | 0m 40s | trunk passed |
| +1 | mvnsite | 1m 5s | trunk passed |
| +1 | mvneclipse | 0m 15s | trunk passed |
| +1 | findbugs | 1m 56s | trunk passed |
| +1 | javadoc | 0m 46s | trunk passed |
| +1 | mvninstall | 1m 0s | the patch passed |
| +1 | compile | 0m 57s | the patch passed |
| +1 | javac | 0m 57s | the patch passed |
| +1 | checkstyle | 0m 40s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 307 unchanged - 2 fixed = 307 total (was 309) |
| +1 | mvnsite | 0m 53s | the patch passed |
| +1 | mvneclipse | 0m 12s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 1m 46s | the patch passed |
| +1 | javadoc | 0m 39s | the patch passed |
| -1 | unit | 68m 57s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 20s | The patch does not generate ASF License warnings. |
| 97m 50s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure090 | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure010 |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12869160/HDFS-11445-005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux cc40f94ddbe2 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c80d3bc |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/19531/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/19531/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/19531/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 17s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 13m 26s | trunk passed |
| +1 | compile | 0m 52s | trunk passed |
| +1 | checkstyle | 0m 41s | trunk passed |
| +1 | mvnsite | 0m 57s | trunk passed |
| +1 | mvneclipse | 0m 15s | trunk passed |
| +1 | findbugs | 1m 45s | trunk passed |
| +1 | javadoc | 0m 46s | trunk passed |
| +1 | mvninstall | 0m 52s | the patch passed |
| +1 | compile | 0m 50s | the patch passed |
| +1 | javac | 0m 50s | the patch passed |
| +1 | checkstyle | 0m 37s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 307 unchanged - 2 fixed = 307 total (was 309) |
| +1 | mvnsite | 0m 53s | the patch passed |
| +1 | mvneclipse | 0m 12s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 1m 52s | the patch passed |
| +1 | javadoc | 0m 38s | the patch passed |
| -1 | unit | 75m 55s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 19s | The patch does not generate ASF License warnings. |
| 102m 29s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure080 | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting | |
| hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | |
| hadoop.hdfs.TestDFSRSDefault10x4StripedOutputStreamWithFailure |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12869160/HDFS-11445-005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 04cdd2706ce6 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c80d3bc |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/19530/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/19530/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/19530/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 20s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 7m 6s | branch-2 passed |
| +1 | compile | 0m 46s | branch-2 passed with JDK v1.8.0_131 |
| +1 | compile | 0m 47s | branch-2 passed with JDK v1.7.0_131 |
| +1 | checkstyle | 0m 32s | branch-2 passed |
| +1 | mvnsite | 0m 56s | branch-2 passed |
| +1 | mvneclipse | 0m 15s | branch-2 passed |
| +1 | findbugs | 2m 8s | branch-2 passed |
| +1 | javadoc | 0m 40s | branch-2 passed with JDK v1.8.0_131 |
| +1 | javadoc | 1m 0s | branch-2 passed with JDK v1.7.0_131 |
| +1 | mvninstall | 0m 49s | the patch passed |
| +1 | compile | 0m 47s | the patch passed with JDK v1.8.0_131 |
| +1 | javac | 0m 47s | the patch passed |
| +1 | compile | 0m 48s | the patch passed with JDK v1.7.0_131 |
| +1 | javac | 0m 48s | the patch passed |
| +1 | checkstyle | 0m 33s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 339 unchanged - 2 fixed = 339 total (was 341) |
| +1 | mvnsite | 0m 58s | the patch passed |
| +1 | mvneclipse | 0m 15s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 2m 26s | the patch passed |
| +1 | javadoc | 0m 41s | the patch passed with JDK v1.8.0_131 |
| +1 | javadoc | 1m 2s | the patch passed with JDK v1.7.0_131 |
| -1 | unit | 52m 24s | hadoop-hdfs in the patch failed with JDK v1.7.0_131. |
| +1 | asflicense | 0m 22s | The patch does not generate ASF License warnings. |
| 144m 39s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_131 Timed out junit tests | org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| org.apache.hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure | |
| JDK v1.7.0_131 Failed junit tests | hadoop.metrics2.sink.TestRollingFileSystemSinkWithSecureHdfs |
| hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:8515d35 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12869161/HDFS-11445-branch-2.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 4b6cb04a38ee 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2 / fe185e2 |
| Default Java | 1.7.0_131 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 |
| findbugs | v3.0.0 |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/19532/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt |
| JDK v1.7.0_131 Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/19532/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/19532/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
test failures are unrelated.Jing Zhao can you please take look branch-2 also..I will commit once after your final review.
Uploading the branch-2.7 patch.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 30s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 8m 11s | branch-2.7 passed |
| +1 | compile | 1m 2s | branch-2.7 passed with JDK v1.8.0_131 |
| +1 | compile | 1m 4s | branch-2.7 passed with JDK v1.7.0_131 |
| +1 | checkstyle | 0m 30s | branch-2.7 passed |
| +1 | mvnsite | 0m 59s | branch-2.7 passed |
| +1 | mvneclipse | 0m 16s | branch-2.7 passed |
| +1 | findbugs | 2m 57s | branch-2.7 passed |
| +1 | javadoc | 0m 58s | branch-2.7 passed with JDK v1.8.0_131 |
| +1 | javadoc | 1m 45s | branch-2.7 passed with JDK v1.7.0_131 |
| +1 | mvninstall | 0m 50s | the patch passed |
| +1 | compile | 0m 55s | the patch passed with JDK v1.8.0_131 |
| +1 | javac | 0m 55s | the patch passed |
| +1 | compile | 0m 58s | the patch passed with JDK v1.7.0_131 |
| +1 | javac | 0m 58s | the patch passed |
| -0 | checkstyle | 0m 27s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 561 unchanged - 1 fixed = 563 total (was 562) |
| +1 | mvnsite | 0m 55s | the patch passed |
| +1 | mvneclipse | 0m 12s | the patch passed |
| -1 | whitespace | 0m 0s | The patch has 3198 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 | whitespace | 1m 22s | The patch 70 line(s) with tabs. |
| +1 | findbugs | 3m 5s | the patch passed |
| +1 | javadoc | 0m 56s | the patch passed with JDK v1.8.0_131 |
| +1 | javadoc | 1m 39s | the patch passed with JDK v1.7.0_131 |
| -1 | unit | 45m 48s | hadoop-hdfs in the patch failed with JDK v1.7.0_131. |
| +1 | asflicense | 0m 18s | The patch does not generate ASF License warnings. |
| 126m 31s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_131 Failed junit tests | hadoop.hdfs.server.namenode.TestFileTruncate |
| hadoop.hdfs.web.TestWebHdfsTokens | |
| hadoop.hdfs.server.datanode.TestBlockScanner | |
| hadoop.hdfs.server.namenode.TestCacheDirectives | |
| hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots | |
| JDK v1.7.0_131 Failed junit tests | hadoop.hdfs.server.namenode.TestDecommissioningStatus |
| hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:c420dfe |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12869205/HDFS-11445-branch-2.7.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 6578bd1243a0 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2.7 / 8a0e5e3 |
| Default Java | 1.7.0_131 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/19537/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
| whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/19537/artifact/patchprocess/whitespace-eol.txt |
| whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/19537/artifact/patchprocess/whitespace-tabs.txt |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/19537/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt |
| JDK v1.7.0_131 Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/19537/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/19537/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
Updated the patch to fix the check-style ('if' construct must use '{}'s,could have fixed while committing.
),import length can be ignored.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 11m 40s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. |
| +1 | mvninstall | 6m 3s | branch-2.7 passed |
| +1 | compile | 0m 59s | branch-2.7 passed with JDK v1.8.0_131 |
| +1 | compile | 1m 1s | branch-2.7 passed with JDK v1.7.0_131 |
| +1 | checkstyle | 0m 29s | branch-2.7 passed |
| +1 | mvnsite | 0m 57s | branch-2.7 passed |
| +1 | mvneclipse | 0m 14s | branch-2.7 passed |
| +1 | findbugs | 2m 57s | branch-2.7 passed |
| +1 | javadoc | 0m 59s | branch-2.7 passed with JDK v1.8.0_131 |
| +1 | javadoc | 1m 45s | branch-2.7 passed with JDK v1.7.0_131 |
| +1 | mvninstall | 0m 53s | the patch passed |
| +1 | compile | 0m 59s | the patch passed with JDK v1.8.0_131 |
| +1 | javac | 0m 59s | the patch passed |
| +1 | compile | 1m 0s | the patch passed with JDK v1.7.0_131 |
| +1 | javac | 1m 0s | the patch passed |
| -0 | checkstyle | 0m 27s | hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 561 unchanged - 1 fixed = 562 total (was 562) |
| +1 | mvnsite | 0m 57s | the patch passed |
| +1 | mvneclipse | 0m 12s | the patch passed |
| -1 | whitespace | 0m 0s | The patch has 3522 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply |
| -1 | whitespace | 1m 27s | The patch 70 line(s) with tabs. |
| +1 | findbugs | 3m 13s | the patch passed |
| +1 | javadoc | 0m 56s | the patch passed with JDK v1.8.0_131 |
| +1 | javadoc | 1m 43s | the patch passed with JDK v1.7.0_131 |
| -1 | unit | 44m 11s | hadoop-hdfs in the patch failed with JDK v1.7.0_131. |
| +1 | asflicense | 0m 18s | The patch does not generate ASF License warnings. |
| 132m 46s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_131 Failed junit tests | hadoop.hdfs.web.TestHttpsFileSystem |
| hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots | |
| JDK v1.7.0_131 Failed junit tests | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:c420dfe |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12869243/HDFS-11445-branch-2.7-002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux 2cc707c48a62 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | branch-2.7 / 8a0e5e3 |
| Default Java | 1.7.0_131 |
| Multi-JDK versions | /usr/lib/jvm/java-8-oracle:1.8.0_131 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_131 |
| findbugs | v3.0.0 |
| checkstyle | https://builds.apache.org/job/PreCommit-HDFS-Build/19541/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt |
| whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/19541/artifact/patchprocess/whitespace-eol.txt |
| whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/19541/artifact/patchprocess/whitespace-tabs.txt |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/19541/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-jdk1.7.0_131.txt |
| JDK v1.7.0_131 Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/19541/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/19541/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
+1, latest patch looks good to me as well.
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11782 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11782/)
HDFS-11445. FSCK shows overall health stauts as corrupt even one replica (brahma: rev 2e41f8803dd46d1bab16c1b206c71be72ea260a1)
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfo.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
Committed to trunk,branch-2,branch-2.8,branch-2.8.1 and branch-2.7. Jing Zhao thanks a lot for great review..Vinayakumar B thanks for additional review.
2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.
For future jira explorers:
while backporting HDFS-11445 into CDH, our internal testing caught a regression in it. After tracing the code, I realized the regression is fixed via HDFS-11755.
Specifically, we found (bogus) missing file warnings when running a Solr example application via Hue.
What's interesting is that the JMX shows MissingBlocks > 0, but there is no missing file names; NameNode Web UI also warnings for missing blocks. But fsck result is healthy.
Steps to reproduce:
1. Install a fresh CDH + CM cluster, 4 nodes (with HDFS-11445).
2. Go to Hue UI, install Solr example.
3. Restart CDH (all services)
For details, this bug seems to happen after writing to a data pipeline. A dfsclient calls FSNamesystem#updatePipeline after it gets acks from the pipeline. However, if FSNamesystem #updatePipeline before DataNodes report IBRs, the block would see zero live replicas (it incorrectly thinks all replicas are stale after the genstamp is updated).
I checked and compared the code in BlockManager#removeStoredBlock and BlockManager#addStoredBlock.
In removeStoredBlock, in addition to removing a DN from a stored block, BlockManager also updates under replication queue (updateNeededReplications); but in addStoredBlock, it does not update under replication queue after adding a DN to a stored block if the file is under construction.
The fix in HDFS-11755 adds an additional check in BlockManager#removeStoredBlock to skip updating under replication queue when the file is under construction, which fixes the problem.
Since HDFS-11755 is committed in branch 2.8 ~ trunk before HDFS-11445, this bug is not seen in these branches. But we should backport HDFS-11755 in branch 2.7 to address the regression.
Uploaded the patch.Kindly Review.