[SPARK-3569] Add metadata field to StructField

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: MLlib, SQL
• Target Version/s: 1.2.0
• Assignee: Xiangrui Meng
• Reporter: Xiangrui Meng
• Shepherd: Michael Armbrust
• Created: 17/Sep/14 19:43
• Updated: 01/Nov/14 21:37
• Resolved: 01/Nov/14 21:37

Description

Want to add a metadata field to StructField that can be used by other applications like ML to embed more information about the column.

case class case class StructField(name: String, dataType: DataType, nullable: Boolean, metadata: Map[String, Any] = Map.empty)

For ML, we can store feature information like categorical/continuous, number categories, category-to-index map, etc.

One question is how to carry over the metadata in query execution. For example:

val features = schemaRDD.select('features)
val featuresDesc = features.schema('features).metadata

Issue Links

• is depended upon by: SPARK-3573 Dataset
• links to: [Github] Pull Request #2701 (mengxr)