[SPARK-25199] InferSchema "all Strings" if one of many CSVs is empty

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Cannot Reproduce
• Affects Version/s: 2.2.1
• Component/s: Input/Output
• Labels: newbie

Environment

I discovered this on AWS Glue, which uses Spark 2.2.1

• Assignee: Unassigned
• Reporter: Neil McGuigan
• Created: 22/Aug/18 18:07
• Updated: 25/Aug/18 21:02
• Resolved: 25/Aug/18 21:02

Description

Spark can load multiple CSV files in one read:

df = spark.read.format("csv").option("header", "true").option("inferSchema", "true").load("/*.csv")

However, if one of these files is empty (though it has a header), Spark will set all column types to "String"

Spark should skip a file for inference if it contains no (non-header) rows

Issue Links

• links to: [Github] Pull Request #22177 (yunjzhang)