[SPARK-2927] Add a conf to configure if we always read Binary columns stored in Parquet as String columns

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.1.0
• Component/s: SQL
• Target Version/s: 1.1.0
• Assignee: Yin Huai
• Reporter: Yin Huai
• Created: 08/Aug/14 17:21
• Updated: 12/Sep/14 21:09
• Resolved: 14/Aug/14 18:05

Description

Based on Parquet spec (https://github.com/Parquet/parquet-format), "strings are stored as byte arrays (binary) with a UTF8 annotation". However, if the data generator does not follow it, we will only read binary values back instead of string values.

Issue Links

• is duplicated by: SPARK-2699 Improve compatibility with parquet file/table
• is related to: SPARK-2699 Improve compatibility with parquet file/table
• links to: [Github] Pull Request #1855 (yhuai)