Details

    • Hide
      Corrected problems with GraphiteSink not having proper format in point tags. Format v1.1 suggests point tags should be delimited with ';' rather than '.' and also appear at the end of the metric name, not in the middle. Also removed risk of point tag value having empty space (will be replaced by '_').
      Show
      Corrected problems with GraphiteSink not having proper format in point tags. Format v1.1 suggests point tags should be delimited with ';' rather than '.' and also appear at the end of the metric name, not in the middle. Also removed risk of point tag value having empty space (will be replaced by '_').
    • metrics graphite
    • Patch

    Description

      org.apache.hadoop.metrics2.GraphiteSink's implementation has certain problems that would make it to generate metrics incorrectly.

      The problem lies with line 77 ~ 84 of the GraphiteSink java:

      for (MetricsTag tag : record.tags()) {
      if (tag.value() != null) {
      metricsPathPrefix.append(".");
      metricsPathPrefix.append(tag.name());
      metricsPathPrefix.append("=");
      metricsPathPrefix.append(tag.value());
      }
      }
      

      It produces point tags having name=value pair in the metrics. However, notice how the tags are added with '.' as its delimiters. Rather than using the '.' character, it should follow the following convention mentioned in the latest graphite doc of using ';' character.

      http://graphite.readthedocs.io/en/latest/tags.html

      Also, the value is not properly being escaped, meaning that if the value has a '.' character in it, it will easily confuse Graphite to accept it as a delimiter, rather than the value. A really good prime example is when the value is a hostname or ip address,

      metrics.example.Hostname=this.is.a.hostname.and.this.is.Metrics 10.0

      In this example, the since the value of the hostname contains '.', it is extremely hard for the receiving end to determine which part is hostname and which part is the rest of the metrics name. A good strategy is to convert any '.' character in the value to be converted to other characters, such as '_'.

      However, the best way would be to follow the latest metrics convention of using ';'

      metrics.example.and.this.is.Metrics;Hostname=this.is.a.hostname 10.0

      Attachments

        1. HADOOP-15230.007.patch
          24 kB
          Howard Yoo

        Issue Links

        Activity

          githubbot ASF GitHub Bot logged work - 06/Feb/21 12:12
          • Time Spent:
            10m
             
            hadoop-yetus commented on pull request #340:
            URL: https://github.com/apache/hadoop/pull/340#issuecomment-774466192


               :confetti_ball: **+1 overall**
               
               
               
               
               
               
               | Vote | Subsystem | Runtime | Logfile | Comment |
               |:----:|----------:|--------:|:--------:|:-------:|
               | +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
               |||| _ Prechecks _ |
               | +1 :green_heart: | dupname | 0m 0s | | No case conflicting files found. |
               | +1 :green_heart: | @author | 0m 0s | | The patch does not contain any @author tags. |
               | +1 :green_heart: | | 0m 0s | [test4tests](test4tests) | The patch appears to include 1 new or modified test files. |
               |||| _ trunk Compile Tests _ |
               | +1 :green_heart: | mvninstall | 32m 21s | | trunk passed |
               | +1 :green_heart: | compile | 20m 41s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
               | +1 :green_heart: | compile | 17m 53s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
               | +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
               | +1 :green_heart: | mvnsite | 1m 32s | | trunk passed |
               | +1 :green_heart: | shadedclient | 16m 6s | | branch has no errors when building and testing our client artifacts. |
               | +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
               | +1 :green_heart: | javadoc | 1m 33s | | trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
               | +0 :ok: | spotbugs | 2m 19s | | Used deprecated FindBugs config; considering switching to SpotBugs. |
               | +1 :green_heart: | findbugs | 2m 17s | | trunk passed |
               | -0 :warning: | patch | 2m 39s | | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. |
               |||| _ Patch Compile Tests _ |
               | +1 :green_heart: | mvninstall | 0m 53s | | the patch passed |
               | +1 :green_heart: | compile | 19m 53s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
               | +1 :green_heart: | javac | 19m 53s | | root-jdkUbuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 generated 0 new + 2021 unchanged - 5 fixed = 2021 total (was 2026) |
               | +1 :green_heart: | compile | 17m 51s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
               | +1 :green_heart: | javac | 17m 51s | | root-jdkPrivateBuild-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 generated 0 new + 1916 unchanged - 5 fixed = 1916 total (was 1921) |
               | +1 :green_heart: | checkstyle | 1m 4s | | hadoop-common-project/hadoop-common: The patch generated 0 new + 1 unchanged - 250 fixed = 1 total (was 251) |
               | +1 :green_heart: | mvnsite | 1m 30s | | the patch passed |
               | +1 :green_heart: | whitespace | 0m 0s | | The patch has no whitespace issues. |
               | +1 :green_heart: | shadedclient | 13m 21s | | patch has no errors when building and testing our client artifacts. |
               | +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 |
               | +1 :green_heart: | javadoc | 1m 35s | | the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
               | +1 :green_heart: | findbugs | 2m 23s | | the patch passed |
               |||| _ Other Tests _ |
               | +1 :green_heart: | unit | 17m 13s | | hadoop-common in the patch passed. |
               | +1 :green_heart: | asflicense | 0m 55s | | The patch does not generate ASF License warnings. |
               | | | 174m 53s | | |
               
               
               | Subsystem | Report/Notes |
               |----------:|:-------------|
               | Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-340/1/artifact/out/Dockerfile |
               | GITHUB PR | https://github.com/apache/hadoop/pull/340 |
               | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle |
               | uname | Linux ac11421f88fa 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6 11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
               | Build tool | maven |
               | Personality | dev-support/bin/hadoop.sh |
               | git revision | trunk / c4918fb2986 |
               | Default Java | Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
               | Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 |
               | Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-340/1/testReport/ |
               | Max. process+thread count | 1296 (vs. ulimit of 5500) |
               | modules | C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common |
               | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-340/1/console |
               | versions | git=2.25.1 maven=3.6.3 findbugs=4.0.6 |
               | Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
               
               
               This message was automatically generated.
               
               


            ----------------------------------------------------------------
            This is an automated message from the Apache Git Service.
            To respond to the message, please log on to GitHub and use the
            URL above to go to the specific comment.

            For queries about this service, please contact Infrastructure at:
            users@infra.apache.org
          githubbot ASF GitHub Bot logged work - 07/Feb/21 22:26
          • Time Spent:
            10m
             
            howardyoo closed pull request #340:
            URL: https://github.com/apache/hadoop/pull/340


               


            ----------------------------------------------------------------
            This is an automated message from the Apache Git Service.
            To respond to the message, please log on to GitHub and use the
            URL above to go to the specific comment.

            For queries about this service, please contact Infrastructure at:
            users@infra.apache.org
          githubbot ASF GitHub Bot logged work - 07/Feb/21 22:29
          • Time Spent:
            10m
             
            howardyoo commented on pull request #340:
            URL: https://github.com/apache/hadoop/pull/340#issuecomment-774779708


               This PR is going nowhere, so I'm closing it, after waiting for more than 2 years now. I first thought this would get merged pretty easily, but somehow this never worked. Maybe I have not done this correctly, but it's hard to know by now. I've been getting email notifications about this sporadically, and was getting annoyed more and more. Now, I guess I can finally say it was a nice try, and good experience. Anyway, I am ready to let this go and forget about it. Good bye!


            ----------------------------------------------------------------
            This is an automated message from the Apache Git Service.
            To respond to the message, please log on to GitHub and use the
            URL above to go to the specific comment.

            For queries about this service, please contact Infrastructure at:
            users@infra.apache.org

          People

            howardyoo Howard Yoo
            howardyoo Howard Yoo

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 0.5h
                0.5h

                Slack