[HDFS-17246] Fix shaded client for building Hadoop on Windows

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: hdfs-client
• Labels: pull-request-available

Environment

Windows 10

• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed

Language

Docker Java bash

• Assignee: Gautham Banasandra
• Reporter: Gautham Banasandra
• Due: 08/Nov/23
• Created: 01/Nov/23 11:38
• Updated: 28/Jan/24 01:13
• Resolved: 01/Nov/23 16:10

Description

Currently, the shaded client Yetus personality in Hadoop fails to build on Windows - https://github.com/apache/hadoop/blob/4c04a6768c0cb3d5081cfa5d84ffb389d92f5805/dev-support/bin/hadoop.sh#L541-L615.

This happens due to the integration test failures in Hadoop client modules - https://github.com/apache/hadoop/tree/4c04a6768c0cb3d5081cfa5d84ffb389d92f5805/hadoop-client-modules/hadoop-client-integration-tests.

There are several issues that need to be addressed in order to get the integration tests working -

1. Set the HADOOP_HOME, needed by the Mini DFS and YARN clusters spawned by the integration tests.
2. Add Hadoop binaries to PATH, so that winutils.exe can be located.
3. Create a new user with Symlink privilege in the Docker image. This is needed for the proper working of Mini YARN cluster, spawned by the integration tests.
4. Fix a bug in DFSUtilClient.java that prevents colon ( : ) in the path. The colon is used a delimiter for the PATH variable while specifying multiple paths. However, this isn't a delimiter in the case of Windows and must be handled appropriately.

Issue Links

• causes: HDFS-17249 Fix TestDFSUtil.testIsValidName() unit test failure
• links to: GitHub Pull Request #5943