[SPARK-2699] Improve compatibility with parquet file/table

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.1.0
• Component/s: SQL
• Target Version/s: 1.2.0
• Assignee: Unassigned
• Reporter: Teng Qiu
• Created: 26/Jul/14 02:07
• Updated: 12/Sep/14 21:09
• Resolved: 12/Sep/14 21:09

Description

after SPARK-2446, the compatibility with parquet file created by old spark release (spark 1.0.x) and by impala (all of versions until now: 1.4.x-cdh5) is broken.

strings in those parquet files are not annotated with UTF8 or are just only ASCII char set (impala doesn't support UTF8 yet)

this ticket aims to add a configuration option or some version check to support those parquet files.

Issue Links

• duplicates: SPARK-2927 Add a conf to configure if we always read Binary columns stored in Parquet as String columns
• relates to: SPARK-2927 Add a conf to configure if we always read Binary columns stored in Parquet as String columns
• links to: [Github] Pull Request #1599 (chutium)

Sub-Tasks

• Hidden files (such as .impala_insert_staging) should be filtered out by sqlContext.parquetFile: Resolved