[SPARK-2179] Public API for DataTypes and Schema

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: SQL
• Target Version/s: 1.1.0
• Assignee: Yin Huai
• Reporter: Michael Armbrust
• Created: 18/Jun/14 08:58
• Updated: 23/Oct/14 02:25
• Resolved: 30/Jul/14 07:20

Description

We want something like the following:

• Expose DataType in the SQL package and lock down all the internal details (TypeTags, etc)
• Programatic API for viewing the schema of an RDD as a StructType
• Method that creates a schema RDD given (RDD[A], StructType, A => Row)

Issue Links

• is blocked by: SPARK-1649 Figure out Nullability semantics for Array elements and Map values
• is duplicated by: SPARK-2276 user should be able to provide schema for table creation
• is related to: SPARK-2360 CSV import to SchemaRDDs
• relates to: SPARK-3988 Public API for DateType support
• links to: [Github] Pull Request #1712 (yhuai)
• links to: [Github] Pull Request #1774 (yhuai)
• links to: Pull request