[SPARK-4588] Add API for feature attributes

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: ML, MLlib
• Target Version/s: 1.4.0
• Assignee: Sean R. Owen
• Reporter: Xiangrui Meng
• Created: 25/Nov/14 05:05
• Updated: 18/Jan/16 20:54
• Resolved: 12/Mar/15 23:35

Description

Feature attributes, e.g., continuous/categorical, feature names, feature dimension, number of categories, number of nonzeros (support) could be useful for ML algorithms.

In SPARK-3569, we added metadata to schema, which can be used to store feature attributes along with the dataset. We need to provide a wrapper over the Metadata class for ML usage.

The design doc is available at https://docs.google.com/document/d/1796XfSzFbZvGWFs0ky99AJhlqkOBRG1O2bUxK2N4Grk/edit?usp=sharing

Issue Links

• blocks: SPARK-5886 Add StringIndexer
• is depended upon by: SPARK-8515 Improve ML attribute API
• is related to: SPARK-12886 Expose params to control how feature names are generated
• links to: [Github] Pull Request #4460 (srowen)
• links to: [Github] Pull Request #4925 (mengxr)