Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11722

Change Datanode file IO profiling sampling to percentage

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: hdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Datanode disk IO profiling sampling is controlled by the setting dfs.datanode.fileio.profiling.sampling.fraction. Instead of fraction, we can a percentage value to make it easier to set.

      1. HDFS-11722.001.patch
        14 kB
        Hanisha Koneru
      2. HDFS-11722.002.patch
        15 kB
        Hanisha Koneru

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
        +1 mvninstall 15m 52s trunk passed
        +1 compile 0m 51s trunk passed
        +1 checkstyle 0m 44s trunk passed
        +1 mvnsite 0m 55s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        -1 findbugs 1m 46s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 0m 41s trunk passed
        +1 mvninstall 0m 51s the patch passed
        +1 compile 0m 45s the patch passed
        +1 javac 0m 45s the patch passed
        -0 checkstyle 0m 39s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 499 unchanged - 0 fixed = 501 total (was 499)
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 findbugs 1m 49s the patch passed
        +1 javadoc 0m 41s the patch passed
        -1 unit 69m 13s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        98m 21s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.namenode.TestMetadataVersionOutput
          hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.tools.TestHdfsConfigFields
          hadoop.hdfs.server.namenode.TestStartup



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ac17dc
        JIRA Issue HDFS-11722
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865641/HDFS-11722.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 4c76066e2009 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 19a7e94
        Default Java 1.8.0_121
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19238/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19238/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19238/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19238/testReport/
        modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19238/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. +1 mvninstall 15m 52s trunk passed +1 compile 0m 51s trunk passed +1 checkstyle 0m 44s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 14s trunk passed -1 findbugs 1m 46s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 0m 41s trunk passed +1 mvninstall 0m 51s the patch passed +1 compile 0m 45s the patch passed +1 javac 0m 45s the patch passed -0 checkstyle 0m 39s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 499 unchanged - 0 fixed = 501 total (was 499) +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 41s the patch passed -1 unit 69m 13s hadoop-hdfs in the patch failed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 98m 21s Reason Tests Failed junit tests hadoop.hdfs.server.namenode.TestMetadataVersionOutput   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.tools.TestHdfsConfigFields   hadoop.hdfs.server.namenode.TestStartup Subsystem Report/Notes Docker Image:yetus/hadoop:0ac17dc JIRA Issue HDFS-11722 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12865641/HDFS-11722.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 4c76066e2009 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 19a7e94 Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19238/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/19238/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/19238/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19238/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19238/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        The patch looks good. Minor comment.

        if (fileIOSamplingPercentage < 0.0001) {
        

        This check should be changed to compare with zero since it's an integer now.

        +1 with that fixed.

        Show
        arpitagarwal Arpit Agarwal added a comment - The patch looks good. Minor comment. if (fileIOSamplingPercentage < 0.0001) { This check should be changed to compare with zero since it's an integer now. +1 with that fixed.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        Also can you please update hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md which still refers to sampling fraction:

        Per-volume metrics contain Datanode Volume IO related statistics. Per-volume
        metrics are off by default. They can be enabled by setting `dfs.datanode
        .fileio.profiling.sampling.fraction` to a fraction between 0.0 and 1.0.
        
        Show
        arpitagarwal Arpit Agarwal added a comment - Also can you please update hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md which still refers to sampling fraction: Per-volume metrics contain Datanode Volume IO related statistics. Per-volume metrics are off by default . They can be enabled by setting `dfs.datanode .fileio.profiling.sampling.fraction` to a fraction between 0.0 and 1.0.
        Hide
        hanishakoneru Hanisha Koneru added a comment -

        Thank you Arpit Agarwal] for reviewing the patch.
        I have addressed your comments and fixed the checkstyle errors in patch v02. Fixed TestHdfsConfigFields test error. Rest of the unit test errors are unrelated.

        Show
        hanishakoneru Hanisha Koneru added a comment - Thank you Arpit Agarwal ] for reviewing the patch. I have addressed your comments and fixed the checkstyle errors in patch v02. Fixed TestHdfsConfigFields test error. Rest of the unit test errors are unrelated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 33s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 5 new or modified test files.
        0 mvndep 0m 16s Maven dependency ordering for branch
        +1 mvninstall 14m 49s trunk passed
        +1 compile 15m 44s trunk passed
        +1 checkstyle 2m 0s trunk passed
        +1 mvnsite 2m 5s trunk passed
        +1 mvneclipse 0m 41s trunk passed
        -1 findbugs 1m 23s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings.
        -1 findbugs 1m 44s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings.
        +1 javadoc 1m 37s trunk passed
        0 mvndep 0m 13s Maven dependency ordering for patch
        +1 mvninstall 1m 27s the patch passed
        +1 compile 14m 4s the patch passed
        +1 javac 14m 4s the patch passed
        +1 checkstyle 2m 0s the patch passed
        +1 mvnsite 2m 2s the patch passed
        +1 mvneclipse 0m 42s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 23s the patch passed
        +1 javadoc 1m 36s the patch passed
        +1 unit 8m 21s hadoop-common in the patch passed.
        -1 unit 64m 40s hadoop-hdfs in the patch failed.
        +1 asflicense 0m 37s The patch does not generate ASF License warnings.
        141m 14s



        Reason Tests
        Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting
          hadoop.hdfs.server.namenode.TestMetadataVersionOutput
          hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
          hadoop.hdfs.server.namenode.TestStartup



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue HDFS-11722
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12866215/HDFS-11722.002.patch
        Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle xml
        uname Linux 0c2c03752b29 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d4631e4
        Default Java 1.8.0_121
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19297/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html
        findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19297/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html
        unit https://builds.apache.org/job/PreCommit-HDFS-Build/19297/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
        Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19297/testReport/
        modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: .
        Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19297/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 33s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 5 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 14m 49s trunk passed +1 compile 15m 44s trunk passed +1 checkstyle 2m 0s trunk passed +1 mvnsite 2m 5s trunk passed +1 mvneclipse 0m 41s trunk passed -1 findbugs 1m 23s hadoop-common-project/hadoop-common in trunk has 19 extant Findbugs warnings. -1 findbugs 1m 44s hadoop-hdfs-project/hadoop-hdfs in trunk has 10 extant Findbugs warnings. +1 javadoc 1m 37s trunk passed 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 1m 27s the patch passed +1 compile 14m 4s the patch passed +1 javac 14m 4s the patch passed +1 checkstyle 2m 0s the patch passed +1 mvnsite 2m 2s the patch passed +1 mvneclipse 0m 42s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 23s the patch passed +1 javadoc 1m 36s the patch passed +1 unit 8m 21s hadoop-common in the patch passed. -1 unit 64m 40s hadoop-hdfs in the patch failed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 141m 14s Reason Tests Failed junit tests hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting   hadoop.hdfs.server.namenode.TestMetadataVersionOutput   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover   hadoop.hdfs.server.namenode.TestStartup Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue HDFS-11722 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12866215/HDFS-11722.002.patch Optional Tests asflicense mvnsite compile javac javadoc mvninstall unit findbugs checkstyle xml uname Linux 0c2c03752b29 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d4631e4 Default Java 1.8.0_121 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19297/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common-warnings.html findbugs https://builds.apache.org/job/PreCommit-HDFS-Build/19297/artifact/patchprocess/branch-findbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html unit https://builds.apache.org/job/PreCommit-HDFS-Build/19297/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/19297/testReport/ modules C: hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs U: . Console output https://builds.apache.org/job/PreCommit-HDFS-Build/19297/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        arpitagarwal Arpit Agarwal added a comment -

        I've committed this to trunk and branch-2 after verifying the test failures are unrelated.

        Thanks for the contribution Hanisha Koneru.

        Show
        arpitagarwal Arpit Agarwal added a comment - I've committed this to trunk and branch-2 after verifying the test failures are unrelated. Thanks for the contribution Hanisha Koneru .
        Hide
        hanishakoneru Hanisha Koneru added a comment - - edited

        Thank you Arpit Agarwal] for committing the patch.

        Show
        hanishakoneru Hanisha Koneru added a comment - - edited Thank you Arpit Agarwal ] for committing the patch.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11682 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11682/)
        HDFS-11722. Change Datanode file IO profiling sampling to percentage. (arp: rev 81092b1f1193cb0d4208960b51ab4ffaddeafe01)

        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeStatusMXBean.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FileIoProvider.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestSlowDiskTracker.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ProfilingFileIoEvents.java
        • (edit) hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMXBean.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeMetrics.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java
        • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestHdfsConfigFields.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11682 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11682/ ) HDFS-11722 . Change Datanode file IO profiling sampling to percentage. (arp: rev 81092b1f1193cb0d4208960b51ab4ffaddeafe01) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DNConf.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeStatusMXBean.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/FileIoProvider.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestSlowDiskTracker.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ProfilingFileIoEvents.java (edit) hadoop-common-project/hadoop-common/src/site/markdown/Metrics.md (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeMXBean.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDataNodeVolumeMetrics.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/Util.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestHdfsConfigFields.java

          People

          • Assignee:
            hanishakoneru Hanisha Koneru
            Reporter:
            hanishakoneru Hanisha Koneru
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development