Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: None
    • Component/s: native
    • Labels:
      None
    • Target Version/s:

      Description

      The initial patch submitted on HADOOP-8223 does not have ability to set file permissions on Windows. This is causing tests to fail. Jira tracks adding support to enable file permissions and ownership changes on Windows

      1. HADOOP-8235-branch-1-win.patch
        124 kB
        Chuan Liu
      2. HADOOP-8235-2-branch-1-win.patch
        121 kB
        Sanjay Radia

        Issue Links

          Activity

          Hide
          Chuan Liu added a comment -

          I have created the following Jiras to track issues Bikas has raised, as well as some known problems.

          HADOOP-8454 Fix the ‘chmod =[perm]’ bug in winutils
          HADOOP-8455 Address user name format on domain joined Windows machines
          HADOOP-8456 Support spaces in user names and group names.
          HADOOP-8457 Address file ownership issue for users in Administrators group on Windows.
          HADOOP-8453 Add unit tests for winutils

          Show
          Chuan Liu added a comment - I have created the following Jiras to track issues Bikas has raised, as well as some known problems. HADOOP-8454 Fix the ‘chmod = [perm] ’ bug in winutils HADOOP-8455 Address user name format on domain joined Windows machines HADOOP-8456 Support spaces in user names and group names. HADOOP-8457 Address file ownership issue for users in Administrators group on Windows. HADOOP-8453 Add unit tests for winutils
          Hide
          Sanjay Radia added a comment -

          Please file jiras to addresses the issues Bikas has raised if needed. In the meantime I will commit the patch to the branch because this patch fixes a large number of tests.

          Show
          Sanjay Radia added a comment - Please file jiras to addresses the issues Bikas has raised if needed. In the meantime I will commit the patch to the branch because this patch fixes a large number of tests.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12530270/HADOOP-8235-2-branch-1-win.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1061//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12530270/HADOOP-8235-2-branch-1-win.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1061//console This message is automatically generated.
          Hide
          Sanjay Radia added a comment -

          Please use dos2unix to convert patch files to unix format - it deals with the ^Ms

          Uploaded converted patch.

          Show
          Sanjay Radia added a comment - Please use dos2unix to convert patch files to unix format - it deals with the ^Ms Uploaded converted patch.
          Hide
          Bikas Saha added a comment -
          +        // If on windows domain, token format is DOMAIN\\user and we want to
          +        // extract only the user name
          +        if (Shell.WINDOWS) {
          +          int i = owner.indexOf('\
          

          Why not remove the domain in the output of winutils itself. Does any other part of the code use the domain information?
          Do you see any issues in user names clashing when the domain is stripped off? If yes, then whats the solution. At least a comment would help.

          +        // FIXME: Group names could have spaces on Windows
          

          Is there a solution in mind? What is the jira tracking this?

                   fs.delete(systemDir, true);
                   if (FileSystem.mkdirs(fs, systemDir, 
                       new FsPermission(SYSTEM_DIR_PERMISSION))) {
          +          if (Shell.WINDOWS) {
          +            // Explicitly set ownership on Windows, as in some scenarios
          +            // Administrators group would end up being the owner what is
          +            // currently not supported by the Hadoop security model.
          +            fs.setOwner(systemDir, getMROwner().getShortUserName(), null);
          +          }
                     break;
                   }
          

          Any issues in doing setOwner regardless of OS? That would make sure expected ownership is always set
          Same for the other places this has been done.

          Show
          Bikas Saha added a comment - + // If on windows domain, token format is DOMAIN\\user and we want to + // extract only the user name + if (Shell.WINDOWS) { + int i = owner.indexOf('\ Why not remove the domain in the output of winutils itself. Does any other part of the code use the domain information? Do you see any issues in user names clashing when the domain is stripped off? If yes, then whats the solution. At least a comment would help. + // FIXME: Group names could have spaces on Windows Is there a solution in mind? What is the jira tracking this? fs.delete(systemDir, true ); if (FileSystem.mkdirs(fs, systemDir, new FsPermission(SYSTEM_DIR_PERMISSION))) { + if (Shell.WINDOWS) { + // Explicitly set ownership on Windows, as in some scenarios + // Administrators group would end up being the owner what is + // currently not supported by the Hadoop security model. + fs.setOwner(systemDir, getMROwner().getShortUserName(), null ); + } break ; } Any issues in doing setOwner regardless of OS? That would make sure expected ownership is always set Same for the other places this has been done.
          Hide
          Chuan Liu added a comment -

          In this patch, we create a native Windows console program to emulate certain Linux command line utilities that are used by Hadoop to set file permissions, ownership, and query file status. The Windows program does not support full command line semantic and functionalities as the equivalent Linux commands. We only provide enough support for the Hadoop usage. The source code can be built on Windows with all versions of Visual Studio 10, including Visual Studio Express 10. The solution file and project file are also generated in Visual Studio 10. (Note for Visual Studio Express: Visual Studio Express does not include compiler for X64 platform. To build on X64 platform, Windows SDK is needed.)

          Corresponding Java code, e.g. Shell.java, are also modified for Hadoop to pick up the correct commands on Windows. There are also a few file permission related test fixes due to different OS behaviors.

          This is just an initial patch for review and collecting feedbacks from community.

          We will later adding tests for the Windows program and make changes based on the feedbacks.

          Show
          Chuan Liu added a comment - In this patch, we create a native Windows console program to emulate certain Linux command line utilities that are used by Hadoop to set file permissions, ownership, and query file status. The Windows program does not support full command line semantic and functionalities as the equivalent Linux commands. We only provide enough support for the Hadoop usage. The source code can be built on Windows with all versions of Visual Studio 10, including Visual Studio Express 10. The solution file and project file are also generated in Visual Studio 10. (Note for Visual Studio Express: Visual Studio Express does not include compiler for X64 platform. To build on X64 platform, Windows SDK is needed.) Corresponding Java code, e.g. Shell.java, are also modified for Hadoop to pick up the correct commands on Windows. There are also a few file permission related test fixes due to different OS behaviors. This is just an initial patch for review and collecting feedbacks from community. We will later adding tests for the Windows program and make changes based on the feedbacks.
          Hide
          Bikas Saha added a comment -

          The proposal is to emulate Hadoop read(r), write(w), execute permissions on top of the Windows ACL based access control. RawLocalFileSystem could be improved to map rwx to Windows permissions and set them for the users and groups using the “icacls” Windows command. Similar commands would need to found for other operations. E.g. we can use “takeown” command for changing the ownership of a file/dir.

          There is no concept of “others” on Windows but that could be emulated using either the built-in “Everyone” or “AuthenticatedUsers” Windows groups. RawLocalFileStatus would map “others” to one of these groups when running on Windows. Alternatively, Hadoop daemons could always assign no permissions to "others" on Windows or make other permissions == group permissions.

          When a user creates a file on Unix then it gets group permissions based on the default group of the user. On Windows, such a file could inherit permissions ACL's from multiple groups (say machine Administrators etc) and we might need to see how the default group permissions can be made to work.

          Lastly, Hadoop uses java setWritable/setReadable etc commands which dont work in some cases for Windows.

          Show
          Bikas Saha added a comment - The proposal is to emulate Hadoop read(r), write(w), execute permissions on top of the Windows ACL based access control. RawLocalFileSystem could be improved to map rwx to Windows permissions and set them for the users and groups using the “icacls” Windows command. Similar commands would need to found for other operations. E.g. we can use “takeown” command for changing the ownership of a file/dir. There is no concept of “others” on Windows but that could be emulated using either the built-in “Everyone” or “AuthenticatedUsers” Windows groups. RawLocalFileStatus would map “others” to one of these groups when running on Windows. Alternatively, Hadoop daemons could always assign no permissions to "others" on Windows or make other permissions == group permissions. When a user creates a file on Unix then it gets group permissions based on the default group of the user. On Windows, such a file could inherit permissions ACL's from multiple groups (say machine Administrators etc) and we might need to see how the default group permissions can be made to work. Lastly, Hadoop uses java setWritable/setReadable etc commands which dont work in some cases for Windows.

            People

            • Assignee:
              Chuan Liu
              Reporter:
              Bikas Saha
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development