Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-13568

Create feature transformer to impute missing values

          Details

          • Type: New Feature
          • Status: Resolved
          • Priority: Minor
          • Resolution: Fixed
          • Affects Version/s: None
          • Fix Version/s: 2.2.0
          • Component/s: ML
          • Labels:
            None

            Description

            It is quite common to encounter missing values in data sets. It would be useful to implement a Transformer that can impute missing data points, similar to e.g. Imputer in scikit-learn.

            Initially, options for imputation could include mean, median and most frequent, but we could add various other approaches. Where possible existing DataFrame code can be used (e.g. for approximate quantiles etc).

              Attachments

                Issue Links

                blocks

                New Feature - A new feature of the product, which has yet to be developed. SPARK-15041 adding mode strategy for ml.feature.Imputer for categorical features

                • Minor - Minor loss of function, or other problem where easy workaround is present.
                • Open

                New Feature - A new feature of the product, which has yet to be developed. SPARK-15040 PySpark impl for ml.feature.Imputer

                • Minor - Minor loss of function, or other problem where easy workaround is present.
                • Resolved

                Documentation - Documentation or Website SPARK-19969 Doc and examples for Imputer

                • Major - Major loss of function.
                • Resolved
                relates to

                Improvement - An improvement or enhancement to an existing feature or task. SPARK-13639 Statistics.colStats(rdd).mean and variance should handle NaN in the input vectors

                • Trivial - Cosmetic problem like misspelt words or misaligned text.
                • Open
                links to

                Pull request #11601 [Github] Pull Request #11601 (hhbyyh)

                  Activity

                    People

                    • Assignee:
                      yuhaoyan yuhao yang
                      Reporter:
                      mlnick Nick Pentreath
                      Shepherd:
                      Nick Pentreath
                    • Votes:
                      1 Vote for this issue
                      Watchers:
                      8 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved: