Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5874 How to improve the current ML pipeline API?
  3. SPARK-8418

Add single- and multi-value support to ML Transformers

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: ML
    • Labels:

      Description

      It would be convenient if all feature transformers supported transforming columns of single values and multiple values, specifically:

      • one column with one value (e.g., type Double)
      • one column with multiple values (e.g., Array[Double] or Vector)

      We could go as far as supporting multiple columns, but that may not be necessary since VectorAssembler could be used to handle that.

      Estimators under ml.feature should also support this.

      This will likely require a short design doc to describe:

      • how input and output columns will be specified
      • schema validation
      • code sharing to reduce duplication

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                josephkb Joseph K. Bradley
              • Votes:
                2 Vote for this issue
                Watchers:
                16 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: