[HADOOP-15461] Improvements over the Hadoop support with Windows

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Giovanni Matteo Fumarola
• Reporter: Giovanni Matteo Fumarola
• Created: 14/May/18 18:27
• Updated: 06/Aug/21 11:12

Description

This Jira tracks the effort to improve the interaction between Hadoop and Windows Server.

• Move away from an external process (winutils.exe) for native code:
  • Replace by native Java APIs (e.g., symlinks);
  • Replace by something like JNI or so;
• Fix the build system to fully leverage cmake instead of msbuild;
• Possible other improvements;
• Memory and handle leaks.

 

I did a quick investigation of the performance of WinUtils in YARN. In average NM calls 4.76 times per second and 65.51 per container.

 

	Requests	Requests/sec	Requests/min	Requests/container
Sum [WinUtils]	135354	4.761	286.160	65.51
[WinUtils] Execute -help	4148	0.145	8.769	2.007
[WinUtils] Execute -ls	2842	0.0999	6.008	1.37
[WinUtils] Execute -systeminfo	9153	0.321	19.35	4.43
[WinUtils] Execute -symlink	115096	4.048	243.33	57.37
[WinUtils] Execute -task isAlive	4115	0.144	8.699	2.05

 Interval: 7 hours, 53 minutes and 48 seconds

Each execution of WinUtils does around 140 IO ops, of which 130 are DDL ops.

This means 666.58 IO ops/second due to WinUtils.

We should start considering to remove WinUtils from Hadoop and creating a JNI interface.

Issue Links

• incorporates: HADOOP-17839 LocalFS to support ability to disable permission get/set; remove need for winutils
• is duplicated by: HDFS-13708 change Files instead of NativeIO

Sub-Tasks

• NativeIO#renameTo on Windows cannot replace an existing file at the destination.: Resolved
• Add additional test cases to cover some corner cases for FileUtil#symlink: Resolved
• Add test cases to cover FileUtil#readLink: Resolved
• Deprecate WinUtils#Symlinks by using native java code: Resolved
• Deprecate Shell#ReadLink by using native java code: Resolved
• Clean up ContainerLaunch and ContainerExecutor pre-HADOOP-15528: Resolved
• Adding support in FileUtil for the creation of directories: Resolved
• Deprecate ContainerLaunch#link by using FileUtil#SymLink: Patch Available
• Missing trimming in readlink in case of protocol: Resolved
• Create a JNI interface to interact with Windows: Open
• [Java] Create a JNI interface to interact with Windows: Open
• [C] Create a JNI interface to interact with Windows: Open
• FileUtil operations don't check for native lib loaded on windows: Open
• winutils.exe is a bug nexus and should be killed with an axe.: Open