Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-14300

GraphiteSink reports metrics containing undesirable whitespaces

Details

    • Bug
    • Status: Patch Available
    • Trivial
    • Resolution: Unresolved
    • 2.5.0, 2.6.0, 2.7.0, 2.7.1, 2.7.2, 2.7.3
    • None
    • metrics
    • None

    Description

      According to the Graphite documentation, the data sent must be in the following format: <metric path> <metric value> <metric timestamp>

      But some tag values that are flattened in the metricpath string sometimes include whitespaces and violate the carbon/graphite format.

      For example, the dirpath enumeration from the datanode :
      FSDatasetState.org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl.Context=FSDatasetState.StorageInfo=FSDataset

      {dirpath='[/data/1/hdfs/data/current, /data/2/hdfs/data/current]'}

      .Hostname=worker1.Capacity 78436466688 1491986609

      Currently, none of the aggregators available in the Graphite ecosystem can handle these malformed metrics. These whitespaces should be replaced by underscores.

      I will be happy to attach a small patch.

      Attachments

        1. HADOOP-14300
          1 kB
          Damien Claveau
        2. HADOOP-14300.001.patch
          9 kB
          Damien Claveau

        Issue Links

          Activity

            damien.claveau Damien Claveau added a comment -

            Patch attached

            damien.claveau Damien Claveau added a comment - Patch attached
            haibochen Haibo Chen added a comment -

            Thanks @damien.claveau for reporting this and providing a patch! Can you also add a unit test in TestGraphiteSink?

            haibochen Haibo Chen added a comment - Thanks @ damien.claveau for reporting this and providing a patch! Can you also add a unit test in TestGraphiteSink?
            damien.claveau Damien Claveau added a comment -

            A couple of things :
            1. There is already a unit test class named org.apache.hadoop.metrics2.impl.TestGraphiteMetrics
            Can I enrich it with my test, or should I rename it TestGraphiteSink to conform to the naming rules, or should I add a new class TestGraphiteSink ?

            2. I think I will have 2 more Jira to open to fully achieve the GraphiteSink.
            The first one is a feature addition to filter out tags from the metric prefix string.
            That would be useful to avoid the situation I have with the tag "StorageInfo".
            I will propose it like in https://issues.apache.org/jira/browse/HADOOP-7507

            The second one will be a fix for the regression described here https://issues.apache.org/jira/browse/HADOOP-11400?focusedCommentId=14977496&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14977496

            Any general advice before I start that work ?

            damien.claveau Damien Claveau added a comment - A couple of things : 1. There is already a unit test class named org.apache.hadoop.metrics2.impl.TestGraphiteMetrics Can I enrich it with my test, or should I rename it TestGraphiteSink to conform to the naming rules, or should I add a new class TestGraphiteSink ? 2. I think I will have 2 more Jira to open to fully achieve the GraphiteSink. The first one is a feature addition to filter out tags from the metric prefix string. That would be useful to avoid the situation I have with the tag "StorageInfo". I will propose it like in https://issues.apache.org/jira/browse/HADOOP-7507 The second one will be a fix for the regression described here https://issues.apache.org/jira/browse/HADOOP-11400?focusedCommentId=14977496&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14977496 Any general advice before I start that work ?

            damien.claveau
            I added you to the contributor list, and assigned the jira to you. Also, submitted the patch for precommit check. Thanks!

            weichiu Wei-Chiu Chuang added a comment - damien.claveau I added you to the contributor list, and assigned the jira to you. Also, submitted the patch for precommit check. Thanks!
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 31s Docker mode activated.
            0 patch 0m 4s The patch file was not named according to hadoop's naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute for instructions.
            +1 @author 0m 0s The patch does not contain any @author tags.
            -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
            +1 mvninstall 13m 38s trunk passed
            +1 compile 15m 43s trunk passed
            +1 checkstyle 0m 36s trunk passed
            +1 mvnsite 1m 4s trunk passed
            +1 mvneclipse 0m 20s trunk passed
            +1 findbugs 1m 26s trunk passed
            +1 javadoc 0m 50s trunk passed
            +1 mvninstall 0m 38s the patch passed
            +1 compile 13m 50s the patch passed
            +1 javac 13m 50s the patch passed
            -0 checkstyle 0m 37s hadoop-common-project/hadoop-common: The patch generated 1 new + 126 unchanged - 1 fixed = 127 total (was 127)
            +1 mvnsite 1m 2s the patch passed
            +1 mvneclipse 0m 20s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 findbugs 1m 38s the patch passed
            +1 javadoc 0m 48s the patch passed
            -1 unit 7m 53s hadoop-common in the patch failed.
            +1 asflicense 0m 34s The patch does not generate ASF License warnings.
            63m 27s



            Reason Tests
            Failed junit tests hadoop.security.TestKDiag



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:612578f
            JIRA Issue HADOOP-14300
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863023/HADOOP-14300
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
            uname Linux 4670b02f8b93 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / 0cab572
            Default Java 1.8.0_121
            findbugs v3.0.0
            checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12095/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
            unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12095/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
            Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12095/testReport/
            modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
            Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12095/console
            Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 31s Docker mode activated. 0 patch 0m 4s The patch file was not named according to hadoop's naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute for instructions. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 38s trunk passed +1 compile 15m 43s trunk passed +1 checkstyle 0m 36s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 20s trunk passed +1 findbugs 1m 26s trunk passed +1 javadoc 0m 50s trunk passed +1 mvninstall 0m 38s the patch passed +1 compile 13m 50s the patch passed +1 javac 13m 50s the patch passed -0 checkstyle 0m 37s hadoop-common-project/hadoop-common: The patch generated 1 new + 126 unchanged - 1 fixed = 127 total (was 127) +1 mvnsite 1m 2s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 38s the patch passed +1 javadoc 0m 48s the patch passed -1 unit 7m 53s hadoop-common in the patch failed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 63m 27s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue HADOOP-14300 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863023/HADOOP-14300 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4670b02f8b93 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0cab572 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12095/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12095/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12095/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12095/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            damien.claveau Damien Claveau added a comment -

            New patch attached with full test coverage (and minor refactoring in the unit test).
            test-patch execution fully succeeded locally.

            damien.claveau Damien Claveau added a comment - New patch attached with full test coverage (and minor refactoring in the unit test). test-patch execution fully succeeded locally.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 16s Docker mode activated.
            +1 @author 0m 0s The patch does not contain any @author tags.
            +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
            +1 mvninstall 13m 16s trunk passed
            +1 compile 16m 25s trunk passed
            +1 checkstyle 0m 38s trunk passed
            +1 mvnsite 1m 4s trunk passed
            +1 mvneclipse 0m 20s trunk passed
            +1 findbugs 1m 28s trunk passed
            +1 javadoc 0m 48s trunk passed
            +1 mvninstall 0m 38s the patch passed
            +1 compile 13m 7s the patch passed
            +1 javac 13m 7s the patch passed
            -0 checkstyle 0m 39s hadoop-common-project/hadoop-common: The patch generated 60 new + 230 unchanged - 26 fixed = 290 total (was 256)
            +1 mvnsite 1m 2s the patch passed
            +1 mvneclipse 0m 20s the patch passed
            +1 whitespace 0m 0s The patch has no whitespace issues.
            +1 findbugs 1m 33s the patch passed
            +1 javadoc 0m 50s the patch passed
            -1 unit 7m 48s hadoop-common in the patch failed.
            +1 asflicense 0m 34s The patch does not generate ASF License warnings.
            62m 45s



            Reason Tests
            Failed junit tests hadoop.security.TestKDiag



            Subsystem Report/Notes
            Docker Image:yetus/hadoop:612578f
            JIRA Issue HADOOP-14300
            JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863429/HADOOP-14300.001.patch
            Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
            uname Linux 0468dc69fa6f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
            Build tool maven
            Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
            git revision trunk / 0cab572
            Default Java 1.8.0_121
            findbugs v3.0.0
            checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12102/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
            unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12102/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
            Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12102/testReport/
            modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
            Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12102/console
            Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 13m 16s trunk passed +1 compile 16m 25s trunk passed +1 checkstyle 0m 38s trunk passed +1 mvnsite 1m 4s trunk passed +1 mvneclipse 0m 20s trunk passed +1 findbugs 1m 28s trunk passed +1 javadoc 0m 48s trunk passed +1 mvninstall 0m 38s the patch passed +1 compile 13m 7s the patch passed +1 javac 13m 7s the patch passed -0 checkstyle 0m 39s hadoop-common-project/hadoop-common: The patch generated 60 new + 230 unchanged - 26 fixed = 290 total (was 256) +1 mvnsite 1m 2s the patch passed +1 mvneclipse 0m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 33s the patch passed +1 javadoc 0m 50s the patch passed -1 unit 7m 48s hadoop-common in the patch failed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 62m 45s Reason Tests Failed junit tests hadoop.security.TestKDiag Subsystem Report/Notes Docker Image:yetus/hadoop:612578f JIRA Issue HADOOP-14300 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863429/HADOOP-14300.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0468dc69fa6f 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0cab572 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/12102/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/12102/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/12102/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/12102/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
            damien.claveau Damien Claveau added a comment -

            Hi Wei-Chiu, thank you for the precommit check.
            I don't think the failed junit tests on hadoop.security.TestKDiag can be related to the patch.

            damien.claveau Damien Claveau added a comment - Hi Wei-Chiu, thank you for the precommit check. I don't think the failed junit tests on hadoop.security.TestKDiag can be related to the patch.
            hadoopqa Hadoop QA added a comment -
            -1 overall



            Vote Subsystem Runtime Comment
            0 reexec 0m 0s Docker mode activated.
            -1 patch 0m 6s HADOOP-14300 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



            This message was automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s HADOOP-14300 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue HADOOP-14300 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12863429/HADOOP-14300.001.patch Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/15685/console Powered by Apache Yetus 0.8.0 http://yetus.apache.org This message was automatically generated.

            People

              damien.claveau Damien Claveau
              damien.claveau Damien Claveau
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: