Details
-
Type:
Improvement
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 3.0.0-alpha4
-
Fix Version/s: 3.0.0-beta1
-
Component/s: metrics
-
Labels:
-
Target Version/s:
-
Hadoop Flags:Incompatible change
-
Release Note:
Description
Going through the split EC/replicated metrics, I think it'd be better to name the replicated-only metrics with "ReplicatedBlocks" rather than "BlocksStat" for clarity. "Stat" is also not a very descriptive name, so remove it for the EC blocks as well. Finally, fix some inconsistencies that were missed earlier.
Issue Links
- relates to
-
HDFS-10999
Introduce separate stats for Replicated and Erasure Coded Blocks apart from the current Aggregated stats
-
- Resolved
-
-
HDFS-12218
Rename split EC / replicated block metrics in BlockManager
-
- Resolved
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
This patch seems to overlap with HDFS-10999.
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | reexec | 0m 21s | Docker mode activated. |
| Prechecks | |||
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| +1 | test4tests | 0m 0s | The patch appears to include 4 new or modified test files. |
| trunk Compile Tests | |||
| +1 | mvninstall | 14m 18s | trunk passed |
| +1 | compile | 0m 49s | trunk passed |
| +1 | checkstyle | 0m 47s | trunk passed |
| +1 | mvnsite | 0m 54s | trunk passed |
| -1 | findbugs | 1m 40s | hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. |
| +1 | javadoc | 0m 40s | trunk passed |
| Patch Compile Tests | |||
| +1 | mvninstall | 0m 49s | the patch passed |
| +1 | compile | 0m 53s | the patch passed |
| +1 | javac | 0m 53s | the patch passed |
| +1 | checkstyle | 0m 47s | the patch passed |
| +1 | mvnsite | 0m 52s | the patch passed |
| +1 | whitespace | 0m 0s | The patch has no whitespace issues. |
| +1 | findbugs | 1m 48s | the patch passed |
| +1 | javadoc | 0m 40s | the patch passed |
| Other Tests | |||
| -1 | unit | 92m 40s | hadoop-hdfs in the patch failed. |
| +1 | asflicense | 0m 23s | The patch does not generate ASF License warnings. |
| 119m 39s |
| Reason | Tests |
|---|---|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| Subsystem | Report/Notes |
|---|---|
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12879208/HDFS-12206.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle |
| uname | Linux e118307f54d9 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 27a1a5f |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs | https://builds.apache.org/job/PreCommit-HDFS-Build/20437/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html |
| unit | https://builds.apache.org/job/PreCommit-HDFS-Build/20437/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt |
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/20437/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/20437/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
The reason why I chose "stat" suffix earlier was to differentiate the new separated metrics from the already existing aggregated metrics and also to avoid overlapping names like "getCorruptReplicaBlocks" vs "getCorruptReplicatedBlocks". Yes, the changes need to be propagated all the way down like LowRedundancyBlocks, other places like ECBlockGroupStats and looks like it will be taken care later.
Code refactoring in patch v01 looks good to me, +1.
Thanks for the review Manoj, committed to trunk!
I filed HDFS-12218 as a follow-on, might need more than that to complete the refactoring.
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12069 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12069/)
HDFS-12206. Rename the split EC / replicated block metrics. (wang: rev 480c8db40c09cd0e25b4d145bc871b70a45d4f50)
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/ECBlockGroupsMBean.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestReconstructStripedBlocks.java
- (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/ReplicatedBlocksMBean.java
- (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/ECBlockGroupsStatsMBean.java
- (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/ReplicatedBlocksStatsMBean.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
- (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
Patch attached. This fixes up the naming of the public interfaces.
These changes should also be propagated down into BlockManager and classes like LowRedundancyBlocks and InvalidateBlocks as well, but that's another big renaming pass. I notice that BM also doesn't respect the same naming conventions as FSN wrt what's an aggregate, replicated-only, EC-only statistic. There's also some inaccuracy wrt "ECBlocks" vs. "ECBlockGroups" in some variable names. I'd like to fix these separately, doing behavior changes especially in separate JIRAs.