[BEAM-8569] Support Hadoop 3 on Beam

• Type: Improvement
• Status: Triage Needed
• Priority: P3
• Resolution: Fixed
• Fix Version/s: 2.27.0
• Component/s: io-java-hadoop-file-system
• Assignee: Piotr Szuberski
• Reporter: Ismaël Mejía
• Created: 06/Nov/19 15:49
• Updated: 13/Apr/23 11:03
• Resolved: 15/May/21 18:01

Description

It seems that Hadoop 3 in production is finally happening. CDH supports it in their latest version and Spark 3 will include support for Hadoop 3 too.

This is an uber ticket to cover the required changes to the codebase to ensure compliance with Hadoop 3.x

Hadoop dependencies in Beam are mostly provided and APIs are until some point compatible, but we may require changes in the CI to test that new changes work both in Hadoop 2 and Hadoop 3 until we decide to remove support for Hadoop 2.

Issue Links

• is related to: BEAM-7093 Support Spark 3 in Spark runner
• is related to: BEAM-4087 Gradle build does not allow to overwrite versions of provided dependencies
• requires: BEAM-8876 Tests to confirm Hadoop version compatibility
• links to: GitHub Pull Request #13354

Sub-Tasks

• Support Hadoop 3.x on Hadoop File System: Resolved
• Support Hadoop 3.x on Hadoop Format: Resolved
• ParquetIO should have Hadoop dependencies as provided: Open
• Support Hadoop 3.x on HCatalog: Open