[SPARK-9407] Parquet shouldn't fail when pushing down predicates over a column whose underlying Parquet type is an ENUM

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.5.0
• Component/s: SQL
• Target Version/s: 1.5.0
• Sprint: Spark 1.5 release
• Assignee: Cheng Lian
• Reporter: Cheng Lian
• Created: 28/Jul/15 16:43
• Updated: 12/Aug/15 12:06
• Resolved: 12/Aug/15 12:06

Description

Spark SQL doesn't have an equivalent data type to Parquet BINARY (ENUM), and always treats it as a UTF-8 encoded StringType. Thus, predicate over a Parquet ENUM column may be pushed down. However, Parquet 1.7.0 and prior versions only support filter push-down optimization for a limited set of data types, and fails the query.

The simplest solution seems to be upgrading parquet-mr to 1.8.1, which fixes this issue via PARQUET-201

Issue Links

• relates to: PARQUET-201 Column with OriginalType INT_8 failed at filtering
• links to: [Github] Pull Request #8107 (liancheng)