Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10959

Adding per disk IO statistics and metrics in DataNode.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      This ticket is opened to support per disk IO statistics in DataNode based on HDFS-10930. The statistics added will make it easier to implement HDFS-4169 "Add per-disk latency metrics to DataNode".

      1. HDFS-10959.00.patch
        28 kB
        Xiaoyu Yao
      2. HDFS-10959.01.patch
        28 kB
        Xiaoyu Yao
      3. HDFS-10959.02.patch
        37 kB
        Xiaoyu Yao
      4. HDFS-10959.03.patch
        37 kB
        Xiaoyu Yao
      5. HDFS-10959.04.patch
        37 kB
        Xiaoyu Yao
      6. HDFS-10959-branch-2.01.patch
        37 kB
        Arpit Agarwal

        Issue Links

          Activity

          Hide
          xyao Xiaoyu Yao added a comment - - edited

          Attach a patch that add per volume io stats and metrics based on ProfilingFileIoEvens.java. It is off by default to minimize the impact on datanode normal IO. When needed, this can be enabled with the following for profiling datanode IO perf.

              <property>
                  <name>dfs.datanode.fileio.events.class</name>     
                  <value>org.apache.hadoop.hdfs.server.datanode.ProfilingFileIoEvents</value>
                  <description>Profiling File IO Events on datanode volumes and expose metrics.</description>
              </property>
          
          Show
          xyao Xiaoyu Yao added a comment - - edited Attach a patch that add per volume io stats and metrics based on ProfilingFileIoEvens.java. It is off by default to minimize the impact on datanode normal IO. When needed, this can be enabled with the following for profiling datanode IO perf. <property> <name>dfs.datanode.fileio.events.class</name> <value>org.apache.hadoop.hdfs.server.datanode.ProfilingFileIoEvents</value> <description>Profiling File IO Events on datanode volumes and expose metrics.</description> </property>
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 8s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 14m 54s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 0m 49s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 44s trunk passed
          +1 javadoc 0m 39s trunk passed
          +1 mvninstall 0m 44s the patch passed
          +1 compile 0m 43s the patch passed
          +1 javac 0m 43s the patch passed
          -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 27 new + 96 unchanged - 0 fixed = 123 total (was 96)
          +1 mvnsite 0m 47s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          -1 findbugs 1m 55s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 javadoc 0m 42s the patch passed
          -1 unit 67m 15s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          94m 0s



          Reason Tests
          FindBugs module:hadoop-hdfs-project/hadoop-hdfs
            Switch statement found in org.apache.hadoop.hdfs.server.datanode.ProfilingFileIoEvents.afterFileIo(FsVolumeSpi, FileIoProvider$OPERATION, long, long) where default case is missing At ProfilingFileIoEvents.java:FileIoProvider$OPERATION, long, long) where default case is missing At ProfilingFileIoEvents.java:[lines 70-81]
            Unused field:DataNodeVolumeMetrics.java
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
            hadoop.hdfs.server.datanode.TestDataNodeMXBean
            hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10959
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843650/HDFS-10959.00.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8bb11a1b60e9 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / f121645
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17878/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17878/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17878/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17878/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17878/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 8s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 14m 54s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 0m 49s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 44s trunk passed +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 44s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed -0 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 27 new + 96 unchanged - 0 fixed = 123 total (was 96) +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. -1 findbugs 1m 55s hadoop-hdfs-project/hadoop-hdfs generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javadoc 0m 42s the patch passed -1 unit 67m 15s hadoop-hdfs in the patch failed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 94m 0s Reason Tests FindBugs module:hadoop-hdfs-project/hadoop-hdfs   Switch statement found in org.apache.hadoop.hdfs.server.datanode.ProfilingFileIoEvents.afterFileIo(FsVolumeSpi, FileIoProvider$OPERATION, long, long) where default case is missing At ProfilingFileIoEvents.java:FileIoProvider$OPERATION, long, long) where default case is missing At ProfilingFileIoEvents.java: [lines 70-81]   Unused field:DataNodeVolumeMetrics.java Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.datanode.TestDataNodeMXBean   hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10959 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843650/HDFS-10959.00.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8bb11a1b60e9 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f121645 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17878/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/17878/artifact/patchprocess/new-findbugs-hadoop-hdfs-project_hadoop-hdfs.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/17878/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17878/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17878/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks Xiaoyu. The patch looks good.

          Minor comments:

          1. FsVolumeImpl#metrics should be final.
          2. We can replace MetadataFileIo with MetadataOperation everwhere, since one call may result in multiple IOs.
          3. Why do we have volumeName.replace(':', '-') in DataNodeVolumeMetrics#create? Is it because the : character is not accepted as part of the metric name?
          4. beforeFileIo and afterFileIo should use sampling. Updating global state for every packet looks inefficient. But we can add the sampling later as the profiling is off by default.
          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks Xiaoyu. The patch looks good. Minor comments: FsVolumeImpl#metrics should be final. We can replace MetadataFileIo with MetadataOperation everwhere, since one call may result in multiple IOs. Why do we have volumeName.replace(':', '-') in DataNodeVolumeMetrics#create ? Is it because the : character is not accepted as part of the metric name? beforeFileIo and afterFileIo should use sampling. Updating global state for every packet looks inefficient. But we can add the sampling later as the profiling is off by default.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks Arpit Agarwal for the review. Update patch to address the feedback.

          Why do we have volumeName.replace(':', '-') in DataNodeVolumeMetrics#create? Is it because the : character is not accepted as part of the metric name?

          Yes. : is not allowed for metric name. That transformation is just to be safe.

          I'm also looking into one of the test failures which seems related.

          Show
          xyao Xiaoyu Yao added a comment - Thanks Arpit Agarwal for the review. Update patch to address the feedback. Why do we have volumeName.replace(':', '-') in DataNodeVolumeMetrics#create? Is it because the : character is not accepted as part of the metric name? Yes. : is not allowed for metric name. That transformation is just to be safe. I'm also looking into one of the test failures which seems related.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 23s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 4 new or modified test files.
          +1 mvninstall 17m 44s trunk passed
          +1 compile 0m 56s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 0m 58s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 59s trunk passed
          +1 javadoc 0m 47s trunk passed
          +1 mvninstall 0m 54s the patch passed
          +1 compile 0m 53s the patch passed
          +1 javac 0m 53s the patch passed
          -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 97 unchanged - 0 fixed = 99 total (was 97)
          +1 mvnsite 1m 6s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 7s the patch passed
          +1 javadoc 0m 48s the patch passed
          -1 unit 95m 52s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          127m 59s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure
            hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10959
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843700/HDFS-10959.01.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8c1a921fc9a8 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / fcbe152
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17883/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17883/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17883/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17883/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 23s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 4 new or modified test files. +1 mvninstall 17m 44s trunk passed +1 compile 0m 56s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 0m 58s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 59s trunk passed +1 javadoc 0m 47s trunk passed +1 mvninstall 0m 54s the patch passed +1 compile 0m 53s the patch passed +1 javac 0m 53s the patch passed -0 checkstyle 0m 29s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 97 unchanged - 0 fixed = 99 total (was 97) +1 mvnsite 1m 6s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 7s the patch passed +1 javadoc 0m 48s the patch passed -1 unit 95m 52s hadoop-hdfs in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 127m 59s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeMultipleRegistrations   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10959 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843700/HDFS-10959.01.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8c1a921fc9a8 3.13.0-96-generic #143-Ubuntu SMP Mon Aug 29 20:15:20 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / fcbe152 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17883/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17883/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17883/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17883/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Update patch that fixed the unit tests.

          Show
          xyao Xiaoyu Yao added a comment - Update patch that fixed the unit tests.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 10s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 15m 6s trunk passed
          +1 compile 0m 52s trunk passed
          +1 checkstyle 0m 32s trunk passed
          +1 mvnsite 1m 4s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 1m 58s trunk passed
          +1 javadoc 0m 43s trunk passed
          +1 mvninstall 0m 57s the patch passed
          +1 compile 0m 54s the patch passed
          +1 javac 0m 54s the patch passed
          -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 110 unchanged - 1 fixed = 113 total (was 111)
          +1 mvnsite 1m 0s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 12s the patch passed
          +1 javadoc 0m 42s the patch passed
          -1 unit 75m 54s hadoop-hdfs in the patch failed.
          +1 asflicense 1m 1s The patch does not generate ASF License warnings.
          105m 23s



          Reason Tests
          Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160
            hadoop.fs.TestSymlinkHdfsFileContext
            hadoop.hdfs.TestMaintenanceState



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10959
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843941/HDFS-10959.02.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux a47285298671 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ef2dd7b
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17899/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/17899/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17899/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17899/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 10s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 15m 6s trunk passed +1 compile 0m 52s trunk passed +1 checkstyle 0m 32s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 1m 58s trunk passed +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 57s the patch passed +1 compile 0m 54s the patch passed +1 javac 0m 54s the patch passed -0 checkstyle 0m 28s hadoop-hdfs-project/hadoop-hdfs: The patch generated 3 new + 110 unchanged - 1 fixed = 113 total (was 111) +1 mvnsite 1m 0s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 12s the patch passed +1 javadoc 0m 42s the patch passed -1 unit 75m 54s hadoop-hdfs in the patch failed. +1 asflicense 1m 1s The patch does not generate ASF License warnings. 105m 23s Reason Tests Failed junit tests hadoop.hdfs.TestDFSStripedOutputStreamWithFailure160   hadoop.fs.TestSymlinkHdfsFileContext   hadoop.hdfs.TestMaintenanceState Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10959 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843941/HDFS-10959.02.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a47285298671 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ef2dd7b Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/17899/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/17899/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17899/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17899/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Update patch to fix the checkstyle issue.

          Show
          xyao Xiaoyu Yao added a comment - Update patch to fix the checkstyle issue.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 12m 34s trunk passed
          +1 compile 0m 46s trunk passed
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 0m 49s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 42s trunk passed
          +1 javadoc 0m 38s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 43s the patch passed
          +1 javac 0m 43s the patch passed
          +1 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 110 unchanged - 1 fixed = 110 total (was 111)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 48s the patch passed
          +1 javadoc 0m 35s the patch passed
          +1 unit 62m 10s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          86m 9s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10959
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843960/HDFS-10959.03.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e653986c6b02 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ef2dd7b
          Default Java 1.8.0_111
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17900/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17900/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 12m 34s trunk passed +1 compile 0m 46s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 49s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 42s trunk passed +1 javadoc 0m 38s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed +1 checkstyle 0m 24s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 110 unchanged - 1 fixed = 110 total (was 111) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 48s the patch passed +1 javadoc 0m 35s the patch passed +1 unit 62m 10s hadoop-hdfs in the patch passed. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 86m 9s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10959 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843960/HDFS-10959.03.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e653986c6b02 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ef2dd7b Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17900/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17900/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jnp Jitendra Nath Pandey added a comment -

          The latest patch looks good to me. +1

          Show
          jnp Jitendra Nath Pandey added a comment - The latest patch looks good to me. +1
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          A few minor comments that can be addressed in a follow up Jira or at commit time.

          1. We should add timeouts to the new unit tests.
          2. Comment line with @Metric annotation can be removed, it looks like a disabled metric. Thanks for the offline explanation about this.
            //@Metric("metadata related operation latency quantiles")
            
          3. Just a suggestion to shorten the metrics descriptions:data related file io flush rate can just be file io flush rate etc. I am fine either way though.
          Show
          arpitagarwal Arpit Agarwal added a comment - A few minor comments that can be addressed in a follow up Jira or at commit time. We should add timeouts to the new unit tests. Comment line with @Metric annotation can be removed, it looks like a disabled metric. Thanks for the offline explanation about this. //@Metric( "metadata related operation latency quantiles" ) Just a suggestion to shorten the metrics descriptions: data related file io flush rate can just be file io flush rate etc. I am fine either way though.
          Hide
          xyao Xiaoyu Yao added a comment - - edited

          Thanks Arpit Agarwal and Jitendra Nath Pandey for the review. Update patch to address the feedback.

          Show
          xyao Xiaoyu Yao added a comment - - edited Thanks Arpit Agarwal and Jitendra Nath Pandey for the review. Update patch to address the feedback.
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Thanks for updating the patch Xiaoyu Yao.

          +1 pending Jenkins.

          Show
          arpitagarwal Arpit Agarwal added a comment - Thanks for updating the patch Xiaoyu Yao . +1 pending Jenkins.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 9s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 6 new or modified test files.
          +1 mvninstall 12m 44s trunk passed
          +1 compile 0m 45s trunk passed
          +1 checkstyle 0m 26s trunk passed
          +1 mvnsite 0m 50s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 41s trunk passed
          +1 javadoc 0m 39s trunk passed
          +1 mvninstall 0m 45s the patch passed
          +1 compile 0m 43s the patch passed
          +1 javac 0m 43s the patch passed
          +1 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 110 unchanged - 1 fixed = 110 total (was 111)
          +1 mvnsite 0m 47s the patch passed
          +1 mvneclipse 0m 10s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 47s the patch passed
          +1 javadoc 0m 37s the patch passed
          +1 unit 62m 32s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 19s The patch does not generate ASF License warnings.
          86m 40s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:a9ad5d6
          JIRA Issue HDFS-10959
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843971/HDFS-10959.04.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 3300371ab6fd 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 3583413
          Default Java 1.8.0_111
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17913/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17913/console
          Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 9s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 6 new or modified test files. +1 mvninstall 12m 44s trunk passed +1 compile 0m 45s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 50s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 39s trunk passed +1 mvninstall 0m 45s the patch passed +1 compile 0m 43s the patch passed +1 javac 0m 43s the patch passed +1 checkstyle 0m 23s hadoop-hdfs-project/hadoop-hdfs: The patch generated 0 new + 110 unchanged - 1 fixed = 110 total (was 111) +1 mvnsite 0m 47s the patch passed +1 mvneclipse 0m 10s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 47s the patch passed +1 javadoc 0m 37s the patch passed +1 unit 62m 32s hadoop-hdfs in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 86m 40s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue HDFS-10959 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12843971/HDFS-10959.04.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3300371ab6fd 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 3583413 Default Java 1.8.0_111 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/17913/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/17913/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xyao Xiaoyu Yao added a comment -

          Thanks all for the review. I've commit the patch to the trunk.

          Show
          xyao Xiaoyu Yao added a comment - Thanks all for the review. I've commit the patch to the trunk.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11015 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11015/)
          HDFS-10959. Adding per disk IO statistics and metrics in DataNode. (xyao: rev fe4ff64a4af4ab5f3ee59ba22ce7db2dec447f7d)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
          • (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/DataNodeVolumeMetrics.java
          • (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeMetrics.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java
          • (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ProfilingFileIoEvents.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11015 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11015/ ) HDFS-10959 . Adding per disk IO statistics and metrics in DataNode. (xyao: rev fe4ff64a4af4ab5f3ee59ba22ce7db2dec447f7d) (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalVolumeImpl.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/DataNodeVolumeMetrics.java (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeMetrics.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/SimulatedFSDataset.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeFailureReporting.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeImpl.java (add) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ProfilingFileIoEvents.java
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Attaching a branch-2 backport patch (the conflicts were trivial).

          This depends on HDFS-11337.

          Show
          arpitagarwal Arpit Agarwal added a comment - Attaching a branch-2 backport patch (the conflicts were trivial). This depends on HDFS-11337 .
          Hide
          arpitagarwal Arpit Agarwal added a comment -

          Pushed to branch-2.

          Show
          arpitagarwal Arpit Agarwal added a comment - Pushed to branch-2.

            People

            • Assignee:
              xyao Xiaoyu Yao
              Reporter:
              xyao Xiaoyu Yao
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development