[SPARK-24289] Unable to Infer schema

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.2.1
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Muraliraja
• Created: 15/May/18 17:46
• Updated: 20/Sep/18 17:40
• Resolved: 16/May/18 01:05

Description

When we run a spark job and when the datatypes during schema inference change and the same is written as parquet files, then when we try to read the data, it fails or mergeSchema fails.

A better way would be to upcast the datatypes to the highest types via a spark option.