Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21818

MultivariateOnlineSummarizer.variance generate negative result

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.2.1, 2.3.0
    • Component/s: ML, MLlib
    • Labels:
      None

      Description

      Because of numerical error, MultivariateOnlineSummarizer.variance is possible to generate negative variance.
      This is a serious bug because many algos in MLLib use stddev computed from sqrt(variance),
      it will generate NaN and crash the whole algorithm.

      we can reproduce this bug use the following code:

          val summarizer1 = (new MultivariateOnlineSummarizer)
            .add(Vectors.dense(3.0), 0.7)
          val summarizer2 = (new MultivariateOnlineSummarizer)
            .add(Vectors.dense(3.0), 0.4)
          val summarizer3 = (new MultivariateOnlineSummarizer)
            .add(Vectors.dense(3.0), 0.5)
          val summarizer4 = (new MultivariateOnlineSummarizer)
            .add(Vectors.dense(3.0), 0.4)
      
          val summarizer = summarizer1
            .merge(summarizer2)
            .merge(summarizer3)
            .merge(summarizer4)
      
          println(summarizer.variance(0))
      

        Attachments

          Activity

            People

            • Assignee:
              WeichenXu123 Weichen Xu
              Reporter:
              WeichenXu123 Weichen Xu
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified