[HADOOP-19018] Release Hadoop 3.4.0

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Shilun Fan
• Reporter: Shilun Fan
• Created: 25/Dec/23 15:52
• Updated: 09/Sep/24 15:59
• Resolved: 19/Mar/24 12:59

Description

Confirmed features to be included in the release:

• Enhanced functionality for YARN Federation.
• Redesigned resource allocation in YARN Capacity Scheduler
• Optimization of HDFS RBF.
• Introduction of fine-grained global locks for DataNodes.
• Improvements in the stability of HDFS EC, and more.
• Fixes for important CVEs.

Issues that need to be addressed in hadoop-3.4.0-RC0 version.

1. confirm the JIRA target version/fix version is 3.4.0 to ensure that the version setting is correct.
2. confirm the highlight of hadoop-3.4.0.
3. backport branch-3.4.0/branch-3.4.

  HADOOP-19040. mvn site commands fails due to MetricsSystem And MetricsSystemImpl changes. 
  YARN-11634. [Addendum] Speed-up TestTimelineClient. 
  MAPREDUCE-7468. [Addendum] Fix TestMapReduceChildJVM unit tests.
  
  Revert HDFS-16016. BPServiceActor to provide new thread to handle IBR.

Issue Links

• is related to: SPARK-41392 Add `bouncy-castle` test dependencies to `sql/core` module for Hadoop 3.4.0
• is related to: SPARK-45393 Upgrade Hadoop to 3.4.0
• is related to: HADOOP-19087 Release Hadoop 3.4.1
• is related to: HADOOP-19036 Standardize Maven Initialization Across Operating Systems
• is related to: HADOOP-19034 Fix Download Maven Url Not Found
• is related to: HADOOP-19038 Improve create-release RUN script
• is related to: HADOOP-19039 Hadoop 3.4.0 Highlight big features and improvements.
• is related to: HADOOP-19040 mvn site commands fails due to MetricsSystem And MetricsSystemImpl changes.
• is related to: HADOOP-19051 Hadoop 3.4.0 Big feature/improvement highlight addendum
• is related to: HADOOP-19065 Update Protocol Buffers installation to 3.21.12
• relates to: HADOOP-19056 Highlight RBF features and improvements targeting version 3.4
• links to: GitHub Pull Request #1

Sub-Tasks

• Use hadoop-thirdparty 1.2.0: Resolved
• prune dependency exports of hadoop-* modules: Resolved
• Add Protobuf Compatibility Notes: Resolved
• Hadoop 3.4.0 release wrap-up: Resolved
• 3.4.0 release documents: Resolved