Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11216

Add remoteBytesRead counter metrics for erasure coding reconstruction task

    Details

    • Type: Task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0-alpha2
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently there is Data Node metrics for how many bytes read during EC reconstruction task. It's better to distinguish the local read and remote read byte counts separately.

        Issue Links

          Activity

          Hide
          Sammi SammiChen added a comment -

          initial patch

          Show
          Sammi SammiChen added a comment - initial patch
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 1m 59s Maven dependency ordering for branch
          +1 mvninstall 14m 32s trunk passed
          +1 compile 10m 26s trunk passed
          +1 checkstyle 1m 38s trunk passed
          +1 mvnsite 2m 3s trunk passed
          +1 mvneclipse 0m 37s trunk passed
          +1 findbugs 3m 30s trunk passed
          +1 javadoc 1m 37s trunk passed
          0 mvndep 0m 17s Maven dependency ordering for patch
          +1 mvninstall 1m 29s the patch passed
          +1 compile 10m 21s the patch passed
          +1 javac 10m 21s the patch passed
          -0 checkstyle 1m 39s root: The patch generated 2 new + 104 unchanged - 0 fixed = 106 total (was 104)
          +1 mvnsite 1m 57s the patch passed
          +1 mvneclipse 0m 36s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 3m 39s the patch passed
          +1 javadoc 1m 37s the patch passed
          +1 unit 8m 9s hadoop-common in the patch passed.
          +1 unit 61m 59s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 34s The patch does not generate ASF License warnings.
          129m 59s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-11216
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843836/HDFS-11216-v1.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 43c75d886933 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ef2dd7b
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17890/artifact/patchprocess/diff-checkstyle-root.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17890/testReport/
          modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17890/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 1m 59s Maven dependency ordering for branch +1 mvninstall 14m 32s trunk passed +1 compile 10m 26s trunk passed +1 checkstyle 1m 38s trunk passed +1 mvnsite 2m 3s trunk passed +1 mvneclipse 0m 37s trunk passed +1 findbugs 3m 30s trunk passed +1 javadoc 1m 37s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 29s the patch passed +1 compile 10m 21s the patch passed +1 javac 10m 21s the patch passed -0 checkstyle 1m 39s root: The patch generated 2 new + 104 unchanged - 0 fixed = 106 total (was 104) +1 mvnsite 1m 57s the patch passed +1 mvneclipse 0m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 39s the patch passed +1 javadoc 1m 37s the patch passed +1 unit 8m 9s hadoop-common in the patch passed. +1 unit 61m 59s hadoop-hdfs in the patch passed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 129m 59s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-11216 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843836/HDFS-11216-v1.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 43c75d886933 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ef2dd7b Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17890/artifact/patchprocess/diff-checkstyle-root.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17890/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17890/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          drankye Kai Zheng added a comment -

          The separation work looks good. One minor, better bytesRead => bytesLocalRead to avoid easy confusing. Accordingly, also take care of the functions. Thanks.

          Show
          drankye Kai Zheng added a comment - The separation work looks good. One minor, better bytesRead => bytesLocalRead to avoid easy confusing. Accordingly, also take care of the functions. Thanks.
          Hide
          Sammi SammiChen added a comment -

          Thanks Kai for reviewing the patch! Let me explain it. The bytesRead is used to hold the total bytes read of ECWorks. remoteBytesRead is used to hold the bytes read from remote DN. So the bytes read from local can be get through bytesRead - remoteBytesRead.

          Show
          Sammi SammiChen added a comment - Thanks Kai for reviewing the patch! Let me explain it. The bytesRead is used to hold the total bytes read of ECWorks. remoteBytesRead is used to hold the bytes read from remote DN. So the bytes read from local can be get through bytesRead - remoteBytesRead .
          Hide
          drankye Kai Zheng added a comment -

          Thanks Sammi for the clarifying. It makes sense. +1 on the patch. Will commit it tomorrow if no other comments.

          Show
          drankye Kai Zheng added a comment - Thanks Sammi for the clarifying. It makes sense. +1 on the patch. Will commit it tomorrow if no other comments.
          Hide
          drankye Kai Zheng added a comment -

          Committed to trunk. Thanks SammiChen for the contribution!

          Show
          drankye Kai Zheng added a comment - Committed to trunk. Thanks SammiChen for the contribution!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11029 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11029/)
          HDFS-11216. Add remoteBytesRead counter metrics for erasure coding (kai.zheng: rev 56a13a6a59cb128cf6fdac78a074faf7e5603967)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeErasureCodingMetrics.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructor.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/MetricsAsserts.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockReconstructor.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockReader.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11029 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11029/ ) HDFS-11216 . Add remoteBytesRead counter metrics for erasure coding (kai.zheng: rev 56a13a6a59cb128cf6fdac78a074faf7e5603967) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeErasureCodingMetrics.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructor.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/MetricsAsserts.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockReconstructor.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedBlockReader.java
          Hide
          Sammi SammiChen added a comment -

          Thanks Kai for review and commit the patch!

          Show
          Sammi SammiChen added a comment - Thanks Kai for review and commit the patch!

            People

            • Assignee:
              Sammi SammiChen
              Reporter:
              Sammi SammiChen
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development