[SPARK-12300] Fix schema inferance on local collections

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 1.6.1, 2.0.0
• Component/s: PySpark, SQL
• Assignee: Holden Karau
• Reporter: Holden Karau
• Created: 12/Dec/15 01:37
• Updated: 04/Jan/16 14:34
• Resolved: 30/Dec/15 19:15

Description

Current schema inferance for local python collections halts as soon as there are no NullTypes. This is different than when we specify a sampling ratio of 1.0 on a distributed collection. This could result in incomplete schema information.

Repro:

input = [{"a": 1}, {"b": "coffee"}]
df = sqlContext.createDataFrame(input)
print df.schema

Discovered while looking at SPARK-2870

Issue Links

• links to: [Github] Pull Request #10275 (holdenk)