Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-12206

Rename the split EC / replicated block metrics

    Details

    • Target Version/s:
    • Hadoop Flags:
      Incompatible change
    • Release Note:
      Hide
      The metrics and MBeans introduced in HDFS-10999 have been renamed for brevity and clarity.
      Show
      The metrics and MBeans introduced in HDFS-10999 have been renamed for brevity and clarity.

      Description

      Going through the split EC/replicated metrics, I think it'd be better to name the replicated-only metrics with "ReplicatedBlocks" rather than "BlocksStat" for clarity. "Stat" is also not a very descriptive name, so remove it for the EC blocks as well. Finally, fix some inconsistencies that were missed earlier.

        Issue Links

          Activity

          Hide
          andrew.wang Andrew Wang added a comment -

          Patch attached. This fixes up the naming of the public interfaces.

          These changes should also be propagated down into BlockManager and classes like LowRedundancyBlocks and InvalidateBlocks as well, but that's another big renaming pass. I notice that BM also doesn't respect the same naming conventions as FSN wrt what's an aggregate, replicated-only, EC-only statistic. There's also some inaccuracy wrt "ECBlocks" vs. "ECBlockGroups" in some variable names. I'd like to fix these separately, doing behavior changes especially in separate JIRAs.

          Show
          andrew.wang Andrew Wang added a comment - Patch attached. This fixes up the naming of the public interfaces. These changes should also be propagated down into BlockManager and classes like LowRedundancyBlocks and InvalidateBlocks as well, but that's another big renaming pass. I notice that BM also doesn't respect the same naming conventions as FSN wrt what's an aggregate, replicated-only, EC-only statistic. There's also some inaccuracy wrt "ECBlocks" vs. "ECBlockGroups" in some variable names. I'd like to fix these separately, doing behavior changes especially in separate JIRAs.
          Hide
          eddyxu Lei (Eddy) Xu added a comment -

          This patch seems to overlap with HDFS-10999.

          Show
          eddyxu Lei (Eddy) Xu added a comment - This patch seems to overlap with HDFS-10999 .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
                trunk Compile Tests
          +1 mvninstall 14m 18s trunk passed
          +1 compile 0m 49s trunk passed
          +1 checkstyle 0m 47s trunk passed
          +1 mvnsite 0m 54s trunk passed
          -1 findbugs 1m 40s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
          +1 javadoc 0m 40s trunk passed
                Patch Compile Tests
          +1 mvninstall 0m 49s the patch passed
          +1 compile 0m 53s the patch passed
          +1 javac 0m 53s the patch passed
          +1 checkstyle 0m 47s the patch passed
          +1 mvnsite 0m 52s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 40s the patch passed
                Other Tests
          -1 unit 92m 40s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          119m 39s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.namenode.ha.TestPipelinesFailover



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue HDFS-12206
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879208/HDFS-12206.001.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e118307f54d9 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 27a1a5f
          Default Java 1.8.0_131
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20437/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/20437/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20437/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20437/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.       trunk Compile Tests +1 mvninstall 14m 18s trunk passed +1 compile 0m 49s trunk passed +1 checkstyle 0m 47s trunk passed +1 mvnsite 0m 54s trunk passed -1 findbugs 1m 40s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 40s trunk passed       Patch Compile Tests +1 mvninstall 0m 49s the patch passed +1 compile 0m 53s the patch passed +1 javac 0m 53s the patch passed +1 checkstyle 0m 47s the patch passed +1 mvnsite 0m 52s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 40s the patch passed       Other Tests -1 unit 92m 40s hadoop-hdfs in the patch failed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 119m 39s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.ha.TestPipelinesFailover Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-12206 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12879208/HDFS-12206.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e118307f54d9 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 27a1a5f Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/20437/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/20437/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/20437/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/20437/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          manojg Manoj Govindassamy added a comment -

          The reason why I chose "stat" suffix earlier was to differentiate the new separated metrics from the already existing aggregated metrics and also to avoid overlapping names like "getCorruptReplicaBlocks" vs "getCorruptReplicatedBlocks". Yes, the changes need to be propagated all the way down like LowRedundancyBlocks, other places like ECBlockGroupStats and looks like it will be taken care later.

          Code refactoring in patch v01 looks good to me, +1.

          Show
          manojg Manoj Govindassamy added a comment - The reason why I chose "stat" suffix earlier was to differentiate the new separated metrics from the already existing aggregated metrics and also to avoid overlapping names like "getCorruptReplicaBlocks" vs "getCorruptReplicatedBlocks". Yes, the changes need to be propagated all the way down like LowRedundancyBlocks , other places like ECBlockGroupStats and looks like it will be taken care later. Code refactoring in patch v01 looks good to me, +1.
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks for the review Manoj, committed to trunk!

          Show
          andrew.wang Andrew Wang added a comment - Thanks for the review Manoj, committed to trunk!
          Hide
          andrew.wang Andrew Wang added a comment -

          I filed HDFS-12218 as a follow-on, might need more than that to complete the refactoring.

          Show
          andrew.wang Andrew Wang added a comment - I filed HDFS-12218 as a follow-on, might need more than that to complete the refactoring.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12069 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12069/)
          HDFS-12206. Rename the split EC / replicated block metrics. (wang: rev 480c8db40c09cd0e25b4d145bc871b70a45d4f50)

          • (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/ECBlockGroupsMBean.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestReconstructStripedBlocks.java
          • (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/ReplicatedBlocksMBean.java
          • (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/ECBlockGroupsStatsMBean.java
          • (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/ReplicatedBlocksStatsMBean.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12069 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12069/ ) HDFS-12206 . Rename the split EC / replicated block metrics. (wang: rev 480c8db40c09cd0e25b4d145bc871b70a45d4f50) (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/ECBlockGroupsMBean.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/metrics/TestNameNodeMetrics.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestReconstructStripedBlocks.java (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/ReplicatedBlocksMBean.java (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/ECBlockGroupsStatsMBean.java (delete) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/ReplicatedBlocksStatsMBean.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestAddStripedBlocks.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java

            People

            • Assignee:
              andrew.wang Andrew Wang
              Reporter:
              andrew.wang Andrew Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development