Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13804

MutableStat mean loses accuracy if add(long, long) is used

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.6.5
    • Fix Version/s: 2.8.0, 2.7.4, 3.0.0-alpha2
    • Component/s: metrics
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Currently if the MutableStat.add(long numSamples, long sum) method is used with a large sample count, the mean that is returned will be very inaccurate. This is a result of using the Welford method for variance calculation, which assumes that each sample is processed on its own, to calculate the mean as well. For variance this is fine, since variance numbers lose meaning if you add many samples at once, but the mean should still be accurate.

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10785 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10785/)
          HADOOP-13804. MutableStat mean loses accuracy if add(long, long) is (zhz: rev 3dbad5d823b8bf61b643dd1057165044138b99e0)

          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/util/SampleStat.java
          • (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableStat.java
          • (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMutableMetrics.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10785 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10785/ ) HADOOP-13804 . MutableStat mean loses accuracy if add(long, long) is (zhz: rev 3dbad5d823b8bf61b643dd1057165044138b99e0) (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/util/SampleStat.java (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/lib/MutableStat.java (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/lib/TestMutableMetrics.java
          Hide
          zhz Zhe Zhang added a comment -

          Committed to trunk~branch-2.7. Thanks Erik for the contribution.

          Show
          zhz Zhe Zhang added a comment - Committed to trunk~branch-2.7. Thanks Erik for the contribution.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 40s trunk passed
          +1 compile 11m 48s trunk passed
          +1 checkstyle 0m 35s trunk passed
          +1 mvnsite 1m 5s trunk passed
          +1 mvneclipse 0m 25s trunk passed
          +1 findbugs 1m 29s trunk passed
          +1 javadoc 0m 51s trunk passed
          +1 mvninstall 0m 36s the patch passed
          +1 compile 9m 57s the patch passed
          +1 javac 9m 57s the patch passed
          -0 checkstyle 0m 31s hadoop-common-project/hadoop-common: The patch generated 2 new + 29 unchanged - 2 fixed = 31 total (was 31)
          +1 mvnsite 1m 4s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 49s the patch passed
          +1 javadoc 0m 54s the patch passed
          +1 unit 8m 31s hadoop-common in the patch passed.
          +1 asflicense 0m 42s The patch does not generate ASF License warnings.
          51m 53s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:e809691
          JIRA Issue HADOOP-13804
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837858/HADOOP-13804.000.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 9aa195bb39aa 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / de3b4aa
          Default Java 1.8.0_111
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11016/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt
          Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11016/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11016/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 40s trunk passed +1 compile 11m 48s trunk passed +1 checkstyle 0m 35s trunk passed +1 mvnsite 1m 5s trunk passed +1 mvneclipse 0m 25s trunk passed +1 findbugs 1m 29s trunk passed +1 javadoc 0m 51s trunk passed +1 mvninstall 0m 36s the patch passed +1 compile 9m 57s the patch passed +1 javac 9m 57s the patch passed -0 checkstyle 0m 31s hadoop-common-project/hadoop-common: The patch generated 2 new + 29 unchanged - 2 fixed = 31 total (was 31) +1 mvnsite 1m 4s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 54s the patch passed +1 unit 8m 31s hadoop-common in the patch passed. +1 asflicense 0m 42s The patch does not generate ASF License warnings. 51m 53s Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue HADOOP-13804 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837858/HADOOP-13804.000.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 9aa195bb39aa 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / de3b4aa Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HADOOP-Build/11016/artifact/patchprocess/diff-checkstyle-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/11016/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/11016/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          zhz Zhe Zhang added a comment -

          Thanks Erik for the fix. The patch LGTM; +1 pending Jenkins.

          Result of the added test without the change:

          java.lang.AssertionError: Bad value for metric TestAvgVal 
          Expected :1.5
          Actual   :1.9995
          
          Show
          zhz Zhe Zhang added a comment - Thanks Erik for the fix. The patch LGTM; +1 pending Jenkins. Result of the added test without the change: java.lang.AssertionError: Bad value for metric TestAvgVal Expected :1.5 Actual :1.9995

            People

            • Assignee:
              xkrogen Erik Krogen
              Reporter:
              xkrogen Erik Krogen
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development