Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.23.0
    • Component/s: metrics
    • Labels:
      None
    • Release Note:
      New metrics2 framework for Hadoop.
    • Tags:
      metrics2

      Description

      The jira tracks the new metrics framework only changes, i.e., it doesn't track the instrumentation changes (and compatibility issues) to existing code.

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #680 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/680/)
          HADOOP-6919. New metrics2 framework. Contributed by Luke Lu.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #680 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk/680/ ) HADOOP-6919 . New metrics2 framework. Contributed by Luke Lu.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #576 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/576/)
          HADOOP-6919. New metrics2 framework. Contributed by Luke Lu.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #576 (See https://builds.apache.org/hudson/job/Hadoop-Common-trunk-Commit/576/ ) HADOOP-6919 . New metrics2 framework. Contributed by Luke Lu.
          Hide
          Arun C Murthy added a comment -

          I just committed this. Thanks Luke!

          Show
          Arun C Murthy added a comment - I just committed this. Thanks Luke!
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12478290/hadoop-6919-metrics2-framework-v6.patch
          against trunk revision 1099633.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 36 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/405//testReport/
          Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/405//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/405//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12478290/hadoop-6919-metrics2-framework-v6.patch against trunk revision 1099633. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 36 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/405//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/405//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-HADOOP-Build/405//console This message is automatically generated.
          Hide
          Luke Lu added a comment -

          When revise for the current trunk (0.23), could you make sure that MetricsContext is backward compatible? It would be good to give them sometime in 0.23 time frame to make the switch. This ensures that HBase can release independently, without tie to Hadoop 0.23 release schedule.

          The existing 0.22+ patches don't remove but deprecate metrics v1 so existing app depending on v1 still works (in a limited way, as they won't see metrics defined in new metrics sources.)

          This is different from the current 0.20.100 branch which removed metrics v1 completely to ensure everything was ported, as the original branch was not targeting external/backward compatibility, which is causing some problems (with trivial workarounds) for dependent projects like Hive that depend on metrics v1.

          Show
          Luke Lu added a comment - When revise for the current trunk (0.23), could you make sure that MetricsContext is backward compatible? It would be good to give them sometime in 0.23 time frame to make the switch. This ensures that HBase can release independently, without tie to Hadoop 0.23 release schedule. The existing 0.22+ patches don't remove but deprecate metrics v1 so existing app depending on v1 still works (in a limited way, as they won't see metrics defined in new metrics sources.) This is different from the current 0.20.100 branch which removed metrics v1 completely to ensure everything was ported, as the original branch was not targeting external/backward compatibility, which is causing some problems (with trivial workarounds) for dependent projects like Hive that depend on metrics v1.
          Hide
          Eric Yang added a comment -

          Thanks for clarified this. I am assuming when Apache Hadoop 0.22 is released, it is going to have MetricsContext v1 instead of v2. HBase depends on the exist hadoop metrics system (v1). This means hbase trunk should work with Apache Hadoop 0.22. When revise for the current trunk (0.23), could you make sure that MetricsContext is backward compatible? It would be good to give them sometime in 0.23 time frame to make the switch. This ensures that HBase can release independently, without tie to Hadoop 0.23 release schedule. Thanks

          Show
          Eric Yang added a comment - Thanks for clarified this. I am assuming when Apache Hadoop 0.22 is released, it is going to have MetricsContext v1 instead of v2. HBase depends on the exist hadoop metrics system (v1). This means hbase trunk should work with Apache Hadoop 0.22. When revise for the current trunk (0.23), could you make sure that MetricsContext is backward compatible? It would be good to give them sometime in 0.23 time frame to make the switch. This ensures that HBase can release independently, without tie to Hadoop 0.23 release schedule. Thanks
          Hide
          Luke Lu added a comment -

          Yeah the v5 of the patch had been applied to Y's 0.22 trunk and the corresponding hdfs/mr code ported since late last year. The patch was originally targeting 0.22. I was told that 0.22 branch is feature frozen as of now. So all the new metrics related patches should probably target 0.23+ releases. I'll rebase the patch against the current trunk (0.23) after the next context switch

          Show
          Luke Lu added a comment - Yeah the v5 of the patch had been applied to Y's 0.22 trunk and the corresponding hdfs/mr code ported since late last year. The patch was originally targeting 0.22. I was told that 0.22 branch is feature frozen as of now. So all the new metrics related patches should probably target 0.23+ releases. I'll rebase the patch against the current trunk (0.23) after the next context switch
          Hide
          Eric Yang added a comment -

          This patch has been around for a while. Does this applies to 0.22 trunk or 0.23 trunk?

          Show
          Eric Yang added a comment - This patch has been around for a while. Does this applies to 0.22 trunk or 0.23 trunk?
          Hide
          Luke Lu added a comment -

          Summary of changes (framework only) vs the jumbo patch for yahoo 0.20.104 (attached in HADOOP-6728 for common/hdfs/mapreduce):

          • Extracted metadata (name and description) for each metric/tag to an interned info object, resulting 33% memory saving per metric/tag (16 bytes vs 24 bytes per metric update (on 64-bit platform) in previous patch (which has the same overhead in the old framework which doesn't even have metric description available.)
          • Simplified metrics source creation with @Metrics (for class) and @Metric (for method and field) annotations (see the metrics2 package doc (package-info.java) and the design doc for examples.)
          Show
          Luke Lu added a comment - Summary of changes (framework only) vs the jumbo patch for yahoo 0.20.104 (attached in HADOOP-6728 for common/hdfs/mapreduce): Extracted metadata (name and description) for each metric/tag to an interned info object, resulting 33% memory saving per metric/tag (16 bytes vs 24 bytes per metric update (on 64-bit platform) in previous patch (which has the same overhead in the old framework which doesn't even have metric description available.) Simplified metrics source creation with @Metrics (for class) and @Metric (for method and field) annotations (see the metrics2 package doc (package-info.java) and the design doc for examples.)
          Hide
          Luke Lu added a comment -

          See package-info info org.apache.hadoop.metrics2 for docs.

          Show
          Luke Lu added a comment - See package-info info org.apache.hadoop.metrics2 for docs.

            People

            • Assignee:
              Luke Lu
              Reporter:
              Luke Lu
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development