[PARQUET-2158] Upgrade Hadoop dependency to version 3.2.0

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.13.0
• Fix Version/s: 1.13.0
• Component/s: parquet-mr
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 13/Jun/22 10:20
• Updated: 23/Jun/24 03:32
• Resolved: 01/Aug/22 09:44

Description

Parquet still builds against Hadoop 2.10. This is very out of date and does not work with java 11, let alone later releases.

Upgrading the dependency to Hadoop 3.2.0 makes the release compatible with java 11, and lines up with active work on HADOOP-18287, Provide a shim library for modern FS APIs

This will significantly speed up access to columnar data, especially in cloud stores.

Issue Links

• relates to: HADOOP-18287 Provide a shim library for modern FS APIs
• relates to: PARQUET-2165 remove deprecated PathGlobPattern and DeprecatedFieldProjectionFilter to compile on hadoop 3.2+
• relates to: HADOOP-12436 GlobPattern regex library has performance issues with wildcard characters