[HADOOP-9637] Adding Native Fstat for Windows as needed by YARN

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0-beta, 3.0.0-alpha1
• Fix Version/s: 2.1.0-beta
• Target Version/s: 2.1.0-beta
• Hadoop Flags: Reviewed
• Assignee: Chuan Liu
• Reporter: Chuan Liu
• Created: 11/Jun/13 01:32
• Updated: 12/May/16 18:27
• Resolved: 18/Jun/13 22:27

Description

In the YARN, nodemanager need to enforce the log file can only be accessed by the owner. At various places, SecureIOUtils.openForRead() was called to enforce this check. We don't have NativeIO.Posix.getFstat() used by SecureIOUtils.openForRead() on Windows, and this make the check fail on Windows. The YARN unit tests TestAggregatedLogFormat.testContainerLogsFileAccess and TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of this.

The JIRA try to provide a Windows implementation of NativeIO.Posix.getFstat().

TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on Windows. The test case try to simulate a situation where first log file is owned by different user (probably symlink) and second one by the user itself. In this situation, the attempt to try to aggregate the logs should fail with the error message "Owner ... for path ... did not match expected owner ...".

The check on file owner happens at AggregatedLogFormat.write() method. The method calls SecureIOUtils.openForRead() to read the log files before writing out to the OutputStream.

SecureIOUtils.openForRead() use NativeIO.Posix.getFstat() to get the file owner and group. We don't have NativeIO.Posix.getFstat() implementation on Windows; thus, the failure.

Issue Links

• is part of: YARN-191 Enhancements to YARN for Windows Server and Windows Azure development and runtime environments