[TEZ-4150] Upgrade to hadoop 3.3

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: László Bodor
• Created: 20/Apr/20 10:49
• Updated: 20/Oct/21 06:39

Description

Hadoop 3.3 is coming, let's check the compatibility once it's released:
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.3+Release

Testing:

# built hadoop trunk locally
mvn clean install -DskipTests -Dhadoop.version=3.4.0-SNAPSHOT

Issue Links

• depends upon: HADOOP-17008 Release Hadoop 3.3.0
• is related to: TEZ-4074 Tez does not run with Hadoop Trunk (3.3.0-snapshot)

Sub-Tasks

• Upgrade to protobuf 3.x and take care of relocated protobuf classes: Resolved
• tez-protobuf: a single module for handling protobuf dependency: Resolved
• ShuffleHandler is not source compatible with Hadoop 3.3.0: Resolved
• Bump hadoop dependency version to 3.3.x: Resolved
• Some jetty jars are not generated in tez-dist: Resolved
• Check dependency version collisions after upgrading to Hadoop 3.3.1: Open