[SPARK-5463] Improve Parquet support (reliability, performance, and error messages)

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.2.0, 1.2.1, 1.2.2, 1.3.0
• Component/s: SQL
• Target Version/s: 1.6.0
• Assignee: Cheng Lian
• Reporter: Cheng Lian
• Created: 28/Jan/15 21:31
• Updated: 03/Nov/15 10:22
• Resolved: 03/Nov/15 10:22

Sub-Tasks

• NPE with new Parquet Filters: Resolved
• And predicates are not properly pushed down: Resolved
• Parquet filter pushdown is not enabled when parquet.task.side.metadata is set to true (default value): Closed
• Data source version of Parquet doesn't push down And filters properly: Resolved
• Cannot use partition columns in where clause when Parquet filter push-down is enabled: Resolved
• Upgrade Parquet to 1.7: Resolved
• Parquet should throw AnalysisException during setup for data type/name related failures: Resolved
• Turn off noisy log output produced by Parquet 1.7.0: Resolved
• Accelerate ParquetRelation2 metadata discovery: Resolved
• Cleanup FilteringParquetRowInputFormat: Resolved
• Better error when saving to parquet with duplicate columns: Resolved
• Test Parquet interoperability and compatibility with other libraries/systems: Resolved
• Parquet : Writing data to S3 very slowly: Resolved
• Turn on Parquet filter push-down by default: Resolved