Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: None
    • Component/s: native
    • Labels:
      None
    • Target Version/s:

      Description

      The initial patch submitted is missing mapping of users to groups. A number of test failures are related to this. The jira tracks adding this support.

      1. HADOOP-8234.branch-1-win.patch
        22 kB
        Bikas Saha
      2. HADOOP-8234.branch-1-win.patch
        11 kB
        Bikas Saha

        Issue Links

          Activity

          Bikas Saha created issue -
          Hide
          Bikas Saha added a comment -

          Hadoop has a GroupMappingServiceProvider interface that is used to get group mappings. The commonly used real implementation of that interface is a Shell based mapper that executes shell commands to obtain this information. There has been some work on LDAP and Active Directory based mappers in the community. The Shell based mapper can be made to work on Windows in the following ways
          1) Map every user to the same group, say “HadoopUsers”. Currently, this is what has been hard coded and it could be changed to be a configurable group name.
          2) Use Windows commands like “net user” to provide the group membership of a given user. This provides both local group memberships and global group memberships. I propose we use this method since it’s a natural extension of the existing feature. It also leverages built-in support on Windows for gathering such information.

          Show
          Bikas Saha added a comment - Hadoop has a GroupMappingServiceProvider interface that is used to get group mappings. The commonly used real implementation of that interface is a Shell based mapper that executes shell commands to obtain this information. There has been some work on LDAP and Active Directory based mappers in the community. The Shell based mapper can be made to work on Windows in the following ways 1) Map every user to the same group, say “HadoopUsers”. Currently, this is what has been hard coded and it could be changed to be a configurable group name. 2) Use Windows commands like “net user” to provide the group membership of a given user. This provides both local group memberships and global group memberships. I propose we use this method since it’s a natural extension of the existing feature. It also leverages built-in support on Windows for gathering such information.
          Hide
          Bikas Saha added a comment -

          posting patch with the shell based user group mappings for windows.
          removing several if(windows) filters because they seem to be running correctly now.

          Show
          Bikas Saha added a comment - posting patch with the shell based user group mappings for windows. removing several if(windows) filters because they seem to be running correctly now.
          Bikas Saha made changes -
          Field Original Value New Value
          Attachment HADOOP-8234.branch-1-win.patch [ 12521414 ]
          Bikas Saha made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 0.24.0, 1.1.0 [ 12317652, 12316501 ] 1.1.0, 0.24.0 [ 12316501, 12317652 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12521414/HADOOP-8234.branch-1-win.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/820//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521414/HADOOP-8234.branch-1-win.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/820//console This message is automatically generated.
          Hide
          Bikas Saha added a comment -

          Patch command applied fine on a different source tree on my machine.

          Show
          Bikas Saha added a comment - Patch command applied fine on a different source tree on my machine.
          Sanjay Radia made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Target Version/s 0.24.0, 1.1.0 [ 12317652, 12316501 ] 1.1.0, 0.24.0 [ 12316501, 12317652 ]
          Hide
          Bikas Saha added a comment -

          The patch does not apply because that works only for trunk.

          I ran the tests on branch-1-win and now 21 additional tests are passing.

            1 org.apache.hadoop.fs.TestFileSystem 
            2 org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser 
            3 org.apache.hadoop.hdfs.security.token.block.TestBlockToken 
            4 org.apache.hadoop.hdfs.server.namenode.TestDFSConcurrentFileOperations 
            5 org.apache.hadoop.hdfs.server.namenode.TestFsck 
            6 org.apache.hadoop.hdfs.TestDFSPermission 
            7 org.apache.hadoop.hdfs.TestDistributedFileSystem 
            8 org.apache.hadoop.hdfs.TestFileAppend4 
            9 org.apache.hadoop.hdfs.TestFileCreation 
           10 org.apache.hadoop.hdfs.TestLeaseRecovery2 
           11 org.apache.hadoop.hdfs.TestQuota 
           12 org.apache.hadoop.hdfs.TestShortCircuitLocalRead 
           13 org.apache.hadoop.hdfs.TestSyncingWriterInterrupted 
           14 org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract 
           15 org.apache.hadoop.mapred.TestMapredSystemDir 
           16 org.apache.hadoop.mapred.TestShuffleJobToken 
           17 org.apache.hadoop.security.TestDistributedCacheAlternateFileSystem 
           18 org.apache.hadoop.security.TestDoAsEffectiveUser 
           19 org.apache.hadoop.security.TestPermission 
           20 org.apache.hadoop.security.TestRefreshUserMappings 
           21 org.apache.hadoop.tools.TestDelegationTokenFetcher
          
          Show
          Bikas Saha added a comment - The patch does not apply because that works only for trunk. I ran the tests on branch-1-win and now 21 additional tests are passing. 1 org.apache.hadoop.fs.TestFileSystem 2 org.apache.hadoop.hdfs.security.TestDelegationTokenForProxyUser 3 org.apache.hadoop.hdfs.security.token.block.TestBlockToken 4 org.apache.hadoop.hdfs.server.namenode.TestDFSConcurrentFileOperations 5 org.apache.hadoop.hdfs.server.namenode.TestFsck 6 org.apache.hadoop.hdfs.TestDFSPermission 7 org.apache.hadoop.hdfs.TestDistributedFileSystem 8 org.apache.hadoop.hdfs.TestFileAppend4 9 org.apache.hadoop.hdfs.TestFileCreation 10 org.apache.hadoop.hdfs.TestLeaseRecovery2 11 org.apache.hadoop.hdfs.TestQuota 12 org.apache.hadoop.hdfs.TestShortCircuitLocalRead 13 org.apache.hadoop.hdfs.TestSyncingWriterInterrupted 14 org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract 15 org.apache.hadoop.mapred.TestMapredSystemDir 16 org.apache.hadoop.mapred.TestShuffleJobToken 17 org.apache.hadoop.security.TestDistributedCacheAlternateFileSystem 18 org.apache.hadoop.security.TestDoAsEffectiveUser 19 org.apache.hadoop.security.TestPermission 20 org.apache.hadoop.security.TestRefreshUserMappings 21 org.apache.hadoop.tools.TestDelegationTokenFetcher
          Hide
          Bikas Saha added a comment -

          In the old patch I had renamed ShellBasedUnixGroupsMapping to ShellBasedUserGroupsMapping. This patch reverts it since merging will be difficult later on.

          Show
          Bikas Saha added a comment - In the old patch I had renamed ShellBasedUnixGroupsMapping to ShellBasedUserGroupsMapping. This patch reverts it since merging will be difficult later on.
          Bikas Saha made changes -
          Attachment HADOOP-8234.branch-1-win.patch [ 12521712 ]
          Hide
          Sanjay Radia added a comment -

          Thanks Bikas, committed to branch-1-win

          Show
          Sanjay Radia added a comment - Thanks Bikas, committed to branch-1-win
          Sanjay Radia made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Target Version/s 1.1.0, 0.24.0 [ 12316501, 12317652 ] HADOOP-1-Windows [ 12320361 ]
          Resolution Fixed [ 1 ]
          Chris Nauroth made changes -
          Link This issue is related to HADOOP-8858 [ HADOOP-8858 ]

            People

            • Assignee:
              Bikas Saha
              Reporter:
              Bikas Saha
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development