Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9637

Adding Native Fstat for Windows as needed by YARN

VotersStop watchingWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 2.1.0-beta, 3.0.0-alpha1
    • 2.1.0-beta
    • None
    • None
    • Reviewed

    Description

      In the YARN, nodemanager need to enforce the log file can only be accessed by the owner. At various places, SecureIOUtils.openForRead() was called to enforce this check. We don't have NativeIO.Posix.getFstat() used by SecureIOUtils.openForRead() on Windows, and this make the check fail on Windows. The YARN unit tests TestAggregatedLogFormat.testContainerLogsFileAccess and TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of this.

      The JIRA try to provide a Windows implementation of NativeIO.Posix.getFstat().

      TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on Windows. The test case try to simulate a situation where first log file is owned by different user (probably symlink) and second one by the user itself. In this situation, the attempt to try to aggregate the logs should fail with the error message "Owner ... for path ... did not match expected owner ...".

      The check on file owner happens at AggregatedLogFormat.write() method. The method calls SecureIOUtils.openForRead() to read the log files before writing out to the OutputStream.

      SecureIOUtils.openForRead() use NativeIO.Posix.getFstat() to get the file owner and group. We don't have NativeIO.Posix.getFstat() implementation on Windows; thus, the failure.

      Attachments

        1. HADOOP-9637-trunk.patch
          14 kB
          Chuan Liu
        2. HADOOP-9637-trunk.2.patch
          15 kB
          Chuan Liu
        3. HADOOP-9637-trunk.3.patch
          15 kB
          Chuan Liu

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            chuanliu Chuan Liu
            chuanliu Chuan Liu
            Votes:
            0 Vote for this issue
            Watchers:
            7 Stop watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment