Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-9637

Adding Native Fstat for Windows as needed by YARN

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0-beta, 3.0.0-alpha1
    • Fix Version/s: 2.1.0-beta
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      In the YARN, nodemanager need to enforce the log file can only be accessed by the owner. At various places, SecureIOUtils.openForRead() was called to enforce this check. We don't have NativeIO.Posix.getFstat() used by SecureIOUtils.openForRead() on Windows, and this make the check fail on Windows. The YARN unit tests TestAggregatedLogFormat.testContainerLogsFileAccess and TestContainerLogsPage.testContainerLogPageAccess fail on Windows because of this.

      The JIRA try to provide a Windows implementation of NativeIO.Posix.getFstat().

      TestAggregatedLogFormat.testContainerLogsFileAccess test case fails on Windows. The test case try to simulate a situation where first log file is owned by different user (probably symlink) and second one by the user itself. In this situation, the attempt to try to aggregate the logs should fail with the error message "Owner ... for path ... did not match expected owner ...".

      The check on file owner happens at AggregatedLogFormat.write() method. The method calls SecureIOUtils.openForRead() to read the log files before writing out to the OutputStream.

      SecureIOUtils.openForRead() use NativeIO.Posix.getFstat() to get the file owner and group. We don't have NativeIO.Posix.getFstat() implementation on Windows; thus, the failure.

        Attachments

        1. HADOOP-9637-trunk.patch
          14 kB
          Chuan Liu
        2. HADOOP-9637-trunk.2.patch
          15 kB
          Chuan Liu
        3. HADOOP-9637-trunk.3.patch
          15 kB
          Chuan Liu

          Issue Links

            Activity

              People

              • Assignee:
                chuanliu Chuan Liu
                Reporter:
                chuanliu Chuan Liu
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: