Uploaded image for project: 'Mahout'
  1. Mahout
  2. MAHOUT-1368

Convert OnlineSummarizer to use the new TDigest

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.9
    • None
    • None

    Description

      The new TDigest provides better accuracy for quartile estimation as well as producing any other quantile you might like. The current quartile estimation of the OnlineSummarizer fails for highly skewed distributions and can't really be extended to provide other quantiles. The TDigest handles all of this.

      Attachments

        1. MAHOUT-1368.patch
          7 kB
          Ted Dunning

        Activity

          tdunning Ted Dunning added a comment -


          Here is a patch with additional skewed test.

          tdunning Ted Dunning added a comment - Here is a patch with additional skewed test.
          smarthi Suneel Marthi added a comment -

          Patch committed to trunk.

          smarthi Suneel Marthi added a comment - Patch committed to trunk.
          hudson Hudson added a comment -

          SUCCESS: Integrated in Mahout-Quality #2352 (See https://builds.apache.org/job/Mahout-Quality/2352/)
          MAHOUT-1368: Convert OnlineSummarizer to use the new TDigest (smarthi: rev 1547911)

          • /mahout/trunk/CHANGELOG
          • /mahout/trunk/examples/src/main/java/org/apache/mahout/clustering/streaming/tools/ClusterQualitySummarizer.java
          • /mahout/trunk/math/src/main/java/org/apache/mahout/math/stats/OnlineSummarizer.java
          • /mahout/trunk/math/src/test/java/org/apache/mahout/math/stats/OnlineSummarizerTest.java
          hudson Hudson added a comment - SUCCESS: Integrated in Mahout-Quality #2352 (See https://builds.apache.org/job/Mahout-Quality/2352/ ) MAHOUT-1368 : Convert OnlineSummarizer to use the new TDigest (smarthi: rev 1547911) /mahout/trunk/CHANGELOG /mahout/trunk/examples/src/main/java/org/apache/mahout/clustering/streaming/tools/ClusterQualitySummarizer.java /mahout/trunk/math/src/main/java/org/apache/mahout/math/stats/OnlineSummarizer.java /mahout/trunk/math/src/test/java/org/apache/mahout/math/stats/OnlineSummarizerTest.java

          People

            smarthi Suneel Marthi
            tdunning Ted Dunning
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: