Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-13568

Create feature transformer to impute missing values

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.2.0
    • Component/s: ML
    • Labels:
      None

      Description

      It is quite common to encounter missing values in data sets. It would be useful to implement a Transformer that can impute missing data points, similar to e.g. Imputer in scikit-learn.

      Initially, options for imputation could include mean, median and most frequent, but we could add various other approaches. Where possible existing DataFrame code can be used (e.g. for approximate quantiles etc).

        Attachments

          Issue Links

          blocks

          New Feature - A new feature of the product, which has yet to be developed. SPARK-15041 adding mode strategy for ml.feature.Imputer for categorical features

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. SPARK-15040 PySpark impl for ml.feature.Imputer

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved

          Documentation - Documentation or Website SPARK-19969 Doc and examples for Imputer

          • Major - Major loss of function.
          • Resolved
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-13639 Statistics.colStats(rdd).mean and variance should handle NaN in the input vectors

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Resolved
          links to

          Pull request #11601 [Github] Pull Request #11601 (hhbyyh)

            Activity

              People

              • Assignee:
                yuhaoyan yuhao yang
                Reporter:
                mlnick Nick Pentreath
                Shepherd:
                Nick Pentreath
              • Votes:
                1 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: