Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17870

ML/MLLIB: ChiSquareSelector based on Statistics.chiSqTest(RDD) is wrong

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.1.0
    • Component/s: ML, MLlib
    • Labels:
      None

      Description

      The method to count ChiSqureTestResult in mllib/feature/ChiSqSelector.scala (line 233) is wrong.

      For feature selection method ChiSquareSelector, it is based on the ChiSquareTestResult.statistic (ChiSqure value) to select the features. It select the features with the largest ChiSqure value. But the Degree of Freedom (df) of ChiSqure value is different in Statistics.chiSqTest(RDD), and for different df, you cannot base on ChiSqure value to select features.

      Because of the wrong method to count ChiSquare value, the feature selection results are strange.
      Take the test suite in ml/feature/ChiSqSelectorSuite.scala as an example:
      If use selectKBest to select: the feature 3 will be selected.
      If use selectFpr to select: feature 1 and 2 will be selected.
      This is strange.

      I use scikit learn to test the same data with the same parameters.
      When use selectKBest to select: feature 1 will be selected.
      When use selectFpr to select: feature 1 and 2 will be selected.
      This result is make sense. because the df of each feature in scikit learn is the same.

      I plan to submit a PR for this problem.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                peng.meng@intel.com Peng Meng
                Reporter:
                peng.meng@intel.com Peng Meng
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: