[SPARK-43197] Clean up the code written for compatibility with Hadoop 2

• Type: Umbrella
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.5.0
• Component/s: Spark Core, SQL, YARN
• Assignee: Unassigned
• Reporter: Yang Jie
• Created: 19/Apr/23 14:00
• Updated: 20/Apr/23 15:50

Description

SPARK-42452 removed support for Hadoop2, we can clean up the code written for compatibility with Hadoop 2 to make it more concise

Issue Links

• links to: [Github] Pull Request #40860 (pan3793)

Sub-Tasks

• Replace reflection w/ direct calling for `ContainerLaunchContext#setTokensConf`: Resolved
• Remove unnecessary serializable wrapper in HadoopFSUtils: Resolved
• Remove workaround for HADOOP-12074: Resolved
• Replace reflection w/ direct calling for Hadoop CallerContext: Resolved
• Remove workaround for MiniKdc's BindException: Resolved
• Remove workaround for FileSinkDesc: Resolved
• Inline `hadoop-client` related properties in `pom.xml`: Resolved
• Resume using enumeration to compare `NodeState.DECOMMISSIONING`: Resolved
• Remove Hadoop 2 reference in docs: Resolved
• Replace reflection w/ direct calling for YARN Resource API: Resolved
• Remove Hadoop2 support in IsolatedClientLoader: Resolved
• Remove `ResourceRequestHelper#isYarnResourceTypesAvailable`: Resolved
• Refactor `ResourceRequestHelper ` to no longer use reflection: Resolved
• Remove check of `isHadoop3`: Resolved
• Unnecessary serialize/deserialize of Path on parallel gather partition stats: Resolved
• Replace reflection w/ direct calling for `SparkHadoopUtil#createFile`: Resolved
• Clean up deprecation hadoop api usage in Yarn module: Resolved
• Directly call `replicate()` instead of reflection in `SparkHadoopUtil#createFile`: Resolved
• Replace the link related to hadoop version 2 with hadoop version 3: Resolved
• Organize hadoop-cloud in standard maven project structure: Resolved