[SPARK-1649] Figure out Nullability semantics for Array elements and Map values

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Done
• Affects Version/s: 1.1.0
• Component/s: SQL
• Assignee: Yin Huai
• Reporter: Andre Schumacher
• Created: 27/Apr/14 08:40
• Updated: 20/Jul/15 01:51
• Resolved: 20/Jul/15 01:51

Description

For the underlying storage layer it would simplify things such as schema conversions, predicate filter determination and such to record in the data type itself whether a column can be nullable. So the DataType type could look like like this:

abstract class DataType(nullable: Boolean = true)

Concrete subclasses could then override the nullable val. Mostly this could be left as the default but when types can be contained in nested types one could optimize for, e.g., arrays with elements that are nullable and those that are not.

Issue Links

• blocks: SPARK-2179 Public API for DataTypes and Schema