Details
-
Type:
Improvement
-
Status: Resolved
-
Priority:
Minor
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 2.8.0, 3.0.0-alpha1
-
Component/s: namenode
-
Labels:None
-
Target Version/s:
-
Hadoop Flags:Reviewed
Description
The BlockManager#removeFromExcessReplicateMap() method accepts a Block which is to remove from excessReplicateMap. However the excessReplicateMap maps a StorageID to the set of BlockInfo that are "extra" for the DataNode of the StorageID. Deleting a sub-class object from a collection provided a base-class object happens to work here.
Alternatively, we can make the removeFromExcessReplicateMap accept a BlockInfo object. As the current call stack is passing BlockInfo object mostly, the code change should be safe.
Issue Links
- is required by
-
HDFS-9442
Move block replication logic from BlockManager to a new class ReplicationManager
-
- Resolved
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 0s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | mvninstall | 9m 15s | trunk passed |
| +1 | compile | 0m 59s | trunk passed with JDK v1.8.0_66 |
| +1 | compile | 0m 56s | trunk passed with JDK v1.7.0_85 |
| +1 | checkstyle | 0m 21s | trunk passed |
| +1 | mvnsite | 1m 9s | trunk passed |
| +1 | mvneclipse | 0m 17s | trunk passed |
| +1 | findbugs | 2m 34s | trunk passed |
| +1 | javadoc | 1m 38s | trunk passed with JDK v1.8.0_66 |
| +1 | javadoc | 2m 30s | trunk passed with JDK v1.7.0_85 |
| +1 | mvninstall | 1m 8s | the patch passed |
| +1 | compile | 1m 11s | the patch passed with JDK v1.8.0_66 |
| +1 | javac | 1m 11s | the patch passed |
| +1 | compile | 0m 57s | the patch passed with JDK v1.7.0_85 |
| +1 | javac | 0m 57s | the patch passed |
| +1 | checkstyle | 0m 21s | the patch passed |
| +1 | mvnsite | 1m 9s | the patch passed |
| +1 | mvneclipse | 0m 16s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 2m 43s | the patch passed |
| +1 | javadoc | 1m 35s | the patch passed with JDK v1.8.0_66 |
| +1 | javadoc | 2m 19s | the patch passed with JDK v1.7.0_85 |
| -1 | unit | 84m 18s | hadoop-hdfs in the patch failed with JDK v1.8.0_66. |
| -1 | unit | 77m 20s | hadoop-hdfs in the patch failed with JDK v1.7.0_85. |
| -1 | asflicense | 0m 25s | Patch generated 58 ASF License warnings. |
| 197m 3s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_66 Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 |
| hadoop.hdfs.TestFileCreationDelete | |
| hadoop.hdfs.shortcircuit.TestShortCircuitCache | |
| hadoop.hdfs.server.datanode.TestDirectoryScanner | |
| hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes | |
| hadoop.hdfs.qjournal.TestSecureNNWithQJM | |
| hadoop.hdfs.server.namenode.TestFileTruncate | |
| hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation | |
| hadoop.hdfs.security.TestDelegationTokenForProxyUser | |
| hadoop.hdfs.server.datanode.TestBlockScanner | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160 | |
| JDK v1.7.0_85 Failed junit tests | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure110 |
| hadoop.hdfs.qjournal.client.TestQuorumJournalManager | |
| hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation | |
| hadoop.hdfs.TestMissingBlocksAlert | |
| hadoop.hdfs.TestDFSUpgradeFromImage |
This message was automatically generated.
Thanks Jing Zhao for reviewing this.
Failing tests are unrelated. TestDirectoryScanner seems flaky.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 0s | Docker mode activated. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | test4tests | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | mvninstall | 11m 43s | trunk passed |
| +1 | compile | 1m 21s | trunk passed with JDK v1.8.0_66 |
| +1 | compile | 1m 2s | trunk passed with JDK v1.7.0_85 |
| +1 | checkstyle | 0m 24s | trunk passed |
| +1 | mvnsite | 1m 17s | trunk passed |
| +1 | mvneclipse | 0m 19s | trunk passed |
| +1 | findbugs | 2m 41s | trunk passed |
| +1 | javadoc | 1m 45s | trunk passed with JDK v1.8.0_66 |
| +1 | javadoc | 2m 48s | trunk passed with JDK v1.7.0_85 |
| +1 | mvninstall | 1m 12s | the patch passed |
| +1 | compile | 1m 18s | the patch passed with JDK v1.8.0_66 |
| +1 | javac | 1m 18s | the patch passed |
| +1 | compile | 1m 1s | the patch passed with JDK v1.7.0_85 |
| +1 | javac | 1m 1s | the patch passed |
| +1 | checkstyle | 0m 24s | the patch passed |
| +1 | mvnsite | 1m 19s | the patch passed |
| +1 | mvneclipse | 0m 17s | the patch passed |
| +1 | whitespace | 0m 0s | Patch has no whitespace issues. |
| +1 | findbugs | 2m 51s | the patch passed |
| +1 | javadoc | 1m 47s | the patch passed with JDK v1.8.0_66 |
| +1 | javadoc | 2m 48s | the patch passed with JDK v1.7.0_85 |
| -1 | unit | 94m 27s | hadoop-hdfs in the patch failed with JDK v1.8.0_66. |
| -1 | unit | 85m 10s | hadoop-hdfs in the patch failed with JDK v1.7.0_85. |
| -1 | asflicense | 0m 28s | Patch generated 56 ASF License warnings. |
| 220m 29s |
| Reason | Tests |
|---|---|
| JDK v1.8.0_66 Failed junit tests | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement | |
| hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes | |
| hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation | |
| hadoop.hdfs.TestDFSStripedOutputStreamWithFailure130 | |
| hadoop.hdfs.server.datanode.TestDirectoryScanner | |
| hadoop.hdfs.server.namenode.TestCheckpoint | |
| hadoop.hdfs.TestEncryptionZones | |
| JDK v1.7.0_85 Failed junit tests | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| hadoop.hdfs.server.datanode.TestDirectoryScanner |
This message was automatically generated.
I've committed this to trunk, branch-2 and branch-2.8. Thanks Mingliang Liu for the contribution!
Thanks Jing Zhao for your review and commit.
FAILURE: Integrated in Hadoop-trunk-Commit #8906 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8906/)
HDFS-9485. Make BlockManager#removeFromExcessReplicateMap accept (jing9: rev 830eb252aaa4fec7ef2ec38cb66f669e8e1ecaa5)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #656 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/656/)
HDFS-9485. Make BlockManager#removeFromExcessReplicateMap accept (jing9: rev 830eb252aaa4fec7ef2ec38cb66f669e8e1ecaa5)
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java
- hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
The patch looks good to me. +1 pending Jenkins.