[SPARK-19411] Remove the metadata used to mark optional columns in merged Parquet schema for filter predicate pushdown

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.2.0
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 31/Jan/17 13:46
• Updated: 03/Feb/17 10:58
• Resolved: 03/Feb/17 10:58

Description

There is a metadata introduced before to mark the optional columns in merged Parquet schema for filter predicate pushdown. As we upgrade to Parquet 1.8.2 which includes the fix for the pushdown of optional columns, we don't need this metadata now.

Issue Links

• links to: [Github] Pull Request #16756 (viirya)