Hadoop Common
  1. Hadoop Common
  2. HADOOP-8943

Support multiple group mapping providers

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.5.0
    • Component/s: security
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Tags:
      Group Mapping

      Description

      Discussed with Natty about LdapGroupMapping, we need to improve it so that:
      1. It's possible to do different group mapping for different users/principals. For example, AD user should go to LdapGroupMapping service for group, but service principals such as hdfs, mapred can still use the default one ShellBasedUnixGroupsMapping;

      2. Multiple ADs can be supported to do LdapGroupMapping;

      3. It's possible to configure what kind of users/principals (regarding domain/realm is an option) should use which group mapping service/mechanism.

      4. It's possible to configure and combine multiple existing mapping providers without writing codes implementing new one.

      1. hadoop-8943-v4.patch
        19 kB
        Kai Zheng
      2. hadoop-8943-v3.patch
        19 kB
        Kai Zheng
      3. hadoop-8943-v2.patch
        19 kB
        Kai Zheng
      4. HADOOP-8943.patch
        27 kB
        Kai Zheng
      5. HADOOP-8943.patch
        11 kB
        Kai Zheng
      6. HADOOP-8943.patch
        27 kB
        Kai Zheng

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          56d 23h 3m 1 Kai Zheng 15/Dec/12 01:43
          Patch Available Patch Available Resolved Resolved
          558d 17h 38m 1 Brandon Li 26/Jun/14 20:21
          Resolved Resolved Closed Closed
          49d 10h 17m 1 Karthik Kambatla (Inactive) 15/Aug/14 06:39
          Karthik Kambatla (Inactive) made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1814 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1814/)
          HADOOP-8943. Support multiple group mapping providers. Contributed by Kai Zheng (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605857)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/CompositeGroupsMapping.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/GroupMappingServiceProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCompositeGroupMapping.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1814 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1814/ ) HADOOP-8943 . Support multiple group mapping providers. Contributed by Kai Zheng (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605857 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/CompositeGroupsMapping.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/GroupMappingServiceProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCompositeGroupMapping.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1787 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1787/)
          HADOOP-8943. Support multiple group mapping providers. Contributed by Kai Zheng (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605857)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/CompositeGroupsMapping.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/GroupMappingServiceProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCompositeGroupMapping.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1787 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1787/ ) HADOOP-8943 . Support multiple group mapping providers. Contributed by Kai Zheng (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605857 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/CompositeGroupsMapping.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/GroupMappingServiceProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCompositeGroupMapping.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #596 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/596/)
          HADOOP-8943. Support multiple group mapping providers. Contributed by Kai Zheng (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605857)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/CompositeGroupsMapping.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/GroupMappingServiceProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCompositeGroupMapping.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #596 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/596/ ) HADOOP-8943 . Support multiple group mapping providers. Contributed by Kai Zheng (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605857 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/CompositeGroupsMapping.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/GroupMappingServiceProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCompositeGroupMapping.java
          Brandon Li made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Resolution Fixed [ 1 ]
          Hide
          Brandon Li added a comment -

          I've committed the patch. Thank you, Kai, for the contribution!

          Show
          Brandon Li added a comment - I've committed the patch. Thank you, Kai, for the contribution!
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #5783 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5783/)
          HADOOP-8943. Support multiple group mapping providers. Contributed by Kai Zheng (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605857)

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/CompositeGroupsMapping.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/GroupMappingServiceProvider.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCompositeGroupMapping.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #5783 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5783/ ) HADOOP-8943 . Support multiple group mapping providers. Contributed by Kai Zheng (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1605857 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/CompositeGroupsMapping.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/GroupMappingServiceProvider.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCompositeGroupMapping.java
          Hide
          Brandon Li added a comment -

          +1. The patch looks good to me. I agree that CommonConfigurationKeysPublic should be moved to the right package.

          Show
          Brandon Li added a comment - +1. The patch looks good to me. I agree that CommonConfigurationKeysPublic should be moved to the right package.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12652593/hadoop-8943-v4.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4174//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4174//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12652593/hadoop-8943-v4.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4174//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4174//console This message is automatically generated.
          Hide
          Kai Zheng added a comment -

          Opened HADOOP-10753 to discuss the concern.

          Show
          Kai Zheng added a comment - Opened HADOOP-10753 to discuss the concern.
          Hide
          Kai Zheng added a comment -

          Brandon, I slightly updated the patch to reference the def in CommonConfigurationKeysPublic. I agree we should use the API but I would think it could be better that CommonConfigurationKeysPublic should be in the right package, like org.apache.hadoop.common instead of org.apache.hadoop.fs, which would help common facility codes like security related avoid coupling with fs stuff. For this I will open a new JIRA to document. Thanks.

          Show
          Kai Zheng added a comment - Brandon, I slightly updated the patch to reference the def in CommonConfigurationKeysPublic. I agree we should use the API but I would think it could be better that CommonConfigurationKeysPublic should be in the right package, like org.apache.hadoop.common instead of org.apache.hadoop.fs, which would help common facility codes like security related avoid coupling with fs stuff. For this I will open a new JIRA to document. Thanks.
          Kai Zheng made changes -
          Attachment hadoop-8943-v4.patch [ 12652593 ]
          Hide
          Kai Zheng added a comment -

          Updated the patch As Brandon suggested.

          Show
          Kai Zheng added a comment - Updated the patch As Brandon suggested.
          Hide
          Brandon Li added a comment -

          Kai Zheng, fs dependency is not a concern since it's already used in several security classes. Also, CommonConfigurationKeysPublic is a stable public class.

          Show
          Brandon Li added a comment - Kai Zheng , fs dependency is not a concern since it's already used in several security classes. Also, CommonConfigurationKeysPublic is a stable public class.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12652178/hadoop-8943-v3.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.ha.TestZKFailoverController

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4155//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4155//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12652178/hadoop-8943-v3.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4155//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4155//console This message is automatically generated.
          Hide
          Kai Zheng added a comment -

          Brandon,

          I updated the patch regarding the comment. As I'm a little not comfortable about introducing the dependency to fs related package for hadoop security facilities by this, I didn't use the CommonConfigurationKeysPublic. Thanks.

          Show
          Kai Zheng added a comment - Brandon, I updated the patch regarding the comment. As I'm a little not comfortable about introducing the dependency to fs related package for hadoop security facilities by this, I didn't use the CommonConfigurationKeysPublic. Thanks.
          Kai Zheng made changes -
          Attachment hadoop-8943-v3.patch [ 12652178 ]
          Hide
          Kai Zheng added a comment -

          Updated the comment as review suggested.

          Show
          Kai Zheng added a comment - Updated the comment as review suggested.
          Hide
          Brandon Li added a comment -

          Kai Zheng, sorry for the late reply. I agree the domain concept introduces unnecessary complexity. If the mapping use case is fairly sophisticated, the user is expected to implemented their own mapping class.

          The updated patch looks good. Some minor comments:
          1. the javadoc of CompositeGroupsMapping#prepareConf is not updated
          2.

          I checked CommonCofigurationKeysPublic, unfortunately it locates in unexpected package.


          I meant CommonConfigurationKeysPublic.HADOOP_SECURITY_GROUP_MAPPING
          The package is package org.apache.hadoop.fs.

          Show
          Brandon Li added a comment - Kai Zheng , sorry for the late reply. I agree the domain concept introduces unnecessary complexity. If the mapping use case is fairly sophisticated, the user is expected to implemented their own mapping class. The updated patch looks good. Some minor comments: 1. the javadoc of CompositeGroupsMapping#prepareConf is not updated 2. I checked CommonCofigurationKeysPublic, unfortunately it locates in unexpected package. I meant CommonConfigurationKeysPublic.HADOOP_SECURITY_GROUP_MAPPING The package is package org.apache.hadoop.fs.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12651957/hadoop-8943-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4139//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4139//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12651957/hadoop-8943-v2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4139//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4139//console This message is automatically generated.
          Hide
          Kai Zheng added a comment -

          Hi Brandon,

          I updated the patch as discussed above.
          1. To simplify the patch, removed the domain support. This would be much safer, and also avoided adding another interface;
          2. Added another property, to indicate whether combining all the groups from all the providers or not.
          hadoop.security.group.mapping.providers.combined: true/false
          3. I checked CommonCofigurationKeysPublic, unfortunately it locates in unexpected package.
          4. Updated the configuration sample in core-default.xml to clarify about how to configure ldap provider in such composite groups mapping provider.

          Would you help review once more? Thanks.

          Show
          Kai Zheng added a comment - Hi Brandon, I updated the patch as discussed above. 1. To simplify the patch, removed the domain support. This would be much safer, and also avoided adding another interface; 2. Added another property, to indicate whether combining all the groups from all the providers or not. hadoop.security.group.mapping.providers.combined: true/false 3. I checked CommonCofigurationKeysPublic, unfortunately it locates in unexpected package. 4. Updated the configuration sample in core-default.xml to clarify about how to configure ldap provider in such composite groups mapping provider. Would you help review once more? Thanks.
          Kai Zheng made changes -
          Attachment hadoop-8943-v2.patch [ 12651957 ]
          Hide
          Kai Zheng added a comment -

          Updated the patch as discussed.

          Show
          Kai Zheng added a comment - Updated the patch as discussed.
          Hide
          Kai Zheng added a comment -

          Hi Brandon,

          Thanks for your review. I will rebase the rather old patch and update it according to your comments.

          1. I re-thought about it, and would think it might be not very necessary and so appropriate to introduce domain here, since we don't have the chance to get any domain attribute from user subject yet due to current authentication mechanism limitation. How about remove domain stuff in the old patch?

          2. The core of this JIRA is to support multiple user groups mapping providers. So how about the logic would you think, assuming there're two providers: shellProvider, adProvider, and a user UserX.
          1) UserX's groups = groups_from_shellProvider + groups_from_adProvider;
          2) UserX's groups = groups_from_shellProvider if any or groups_from_adProvider if any. If both are not empty, then the first provider listed the hadoop.security.group.mapping.providers will win.

          Which one would you think is better?

          3. About how to configure the providers, I extracted some properties from the patch as example. Can this work?
          hadoop.security.group.mapping.providers: shell4services,ad4usersX,ad4usersY
          hadoop.security.group.mapping.provider.shell4services: org.apache.hadoop.security.ShellBasedUnixGroupsMapping
          hadoop.security.group.mapping.provider.ad4usersX: org.apache.hadoop.security.LdapGroupsMapping
          ...
          hadoop.security.group.mapping.provider.ad4usersX.ldap.url: adX.example.com
          hadoop.security.group.mapping.provider.ad4usersY.ldap.url: adY.example.com
          ...

          Show
          Kai Zheng added a comment - Hi Brandon, Thanks for your review. I will rebase the rather old patch and update it according to your comments. 1. I re-thought about it, and would think it might be not very necessary and so appropriate to introduce domain here, since we don't have the chance to get any domain attribute from user subject yet due to current authentication mechanism limitation. How about remove domain stuff in the old patch? 2. The core of this JIRA is to support multiple user groups mapping providers. So how about the logic would you think, assuming there're two providers: shellProvider, adProvider, and a user UserX. 1) UserX's groups = groups_from_shellProvider + groups_from_adProvider; 2) UserX's groups = groups_from_shellProvider if any or groups_from_adProvider if any. If both are not empty, then the first provider listed the hadoop.security.group.mapping.providers will win. Which one would you think is better? 3. About how to configure the providers, I extracted some properties from the patch as example. Can this work? hadoop.security.group.mapping.providers: shell4services,ad4usersX,ad4usersY hadoop.security.group.mapping.provider.shell4services: org.apache.hadoop.security.ShellBasedUnixGroupsMapping hadoop.security.group.mapping.provider.ad4usersX: org.apache.hadoop.security.LdapGroupsMapping ... hadoop.security.group.mapping.provider.ad4usersX.ldap.url: adX.example.com hadoop.security.group.mapping.provider.ad4usersY.ldap.url: adY.example.com ...
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12561104/HADOOP-8943.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4076//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12561104/HADOOP-8943.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4076//console This message is automatically generated.
          Hide
          Brandon Li added a comment -

          The patch looks pretty nice. A few comments:

          1. for the change in the public interface GroupMappingServiceProvider.java. We can use the same config key already defined in CommonCofigurationKeysPublic instead of defining a new one:

          + public static final String GROUP_MAPPING_CONFIG_PREFIX = "hadoop.security.group.mapping";

          2. when multiple domains are configured and LdapGroupsMapping is used multiple times for all the domains(as in the example given in core-site.xml). We may also need multiple ladap password and password file config keys for different ldap servers. Currently all ldap servers use the same configuration in hadoop.security.group.mapping.ldap.ssl.keystore.password.file and hadoop.security.group.mapping.ldap.bind.password.file.

          3. you may want to change MAPPING_PROVIDERS_KEY to MAPPING_PROVIDERS_CONFIG_PREFIX to be consistent with others, like MAPPING_PROVIDER_CONFIG_PREFIX and GROUP_MAPPING_CONFIG_PREFIX.

          4. please add java doc for domain param in Groups#getGroups, and doGetGroups

          The patch needs to be rebased too.

          Show
          Brandon Li added a comment - The patch looks pretty nice. A few comments: 1. for the change in the public interface GroupMappingServiceProvider.java. We can use the same config key already defined in CommonCofigurationKeysPublic instead of defining a new one: + public static final String GROUP_MAPPING_CONFIG_PREFIX = "hadoop.security.group.mapping"; 2. when multiple domains are configured and LdapGroupsMapping is used multiple times for all the domains(as in the example given in core-site.xml). We may also need multiple ladap password and password file config keys for different ldap servers. Currently all ldap servers use the same configuration in hadoop.security.group.mapping.ldap.ssl.keystore.password.file and hadoop.security.group.mapping.ldap.bind.password.file. 3. you may want to change MAPPING_PROVIDERS_KEY to MAPPING_PROVIDERS_CONFIG_PREFIX to be consistent with others, like MAPPING_PROVIDER_CONFIG_PREFIX and GROUP_MAPPING_CONFIG_PREFIX. 4. please add java doc for domain param in Groups#getGroups, and doGetGroups The patch needs to be rebased too.
          Arun C Murthy made changes -
          Fix Version/s 2.5.0 [ 12326263 ]
          Fix Version/s 2.4.0 [ 12326144 ]
          Arun C Murthy made changes -
          Fix Version/s 2.4.0 [ 12326144 ]
          Fix Version/s 2.3.0 [ 12325254 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12561104/HADOOP-8943.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3492//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12561104/HADOOP-8943.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3492//console This message is automatically generated.
          Arun C Murthy made changes -
          Fix Version/s 2.3.0 [ 12325254 ]
          Fix Version/s 2.4.0 [ 12324587 ]
          Arun C Murthy made changes -
          Fix Version/s 2.3.0 [ 12324587 ]
          Fix Version/s 2.1.0-beta [ 12324030 ]
          Kai Zheng made changes -
          Assignee Kai Zheng [ drankye ]
          Hide
          Andrew Grande added a comment -

          Hi, any action on this patch? All checkpoints green. This is the major step in supporting access control lists on Hadoop.

          Show
          Andrew Grande added a comment - Hi, any action on this patch? All checkpoints green. This is the major step in supporting access control lists on Hadoop.
          Arun C Murthy made changes -
          Fix Version/s 2.0.4-beta [ 12324030 ]
          Fix Version/s 2.0.3-alpha [ 12323273 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12561104/HADOOP-8943.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1879//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1879//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12561104/HADOOP-8943.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1879//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1879//console This message is automatically generated.
          Kai Zheng made changes -
          Attachment HADOOP-8943.patch [ 12561104 ]
          Hide
          Kai Zheng added a comment -

          Updated the patch with missed files.

          Show
          Kai Zheng added a comment - Updated the patch with missed files.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12561082/HADOOP-8943.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1877//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12561082/HADOOP-8943.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1877//console This message is automatically generated.
          Kai Zheng made changes -
          Link This issue is related to HADOOP-4656 [ HADOOP-4656 ]
          Kai Zheng made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Target Version/s 2.0.3-alpha [ 12323273 ]
          Tags Ldap AD GroupMapping Group Mapping
          Kai Zheng made changes -
          Attachment HADOOP-8943.patch [ 12561082 ]
          Hide
          Kai Zheng added a comment -

          Updated patch with sync.

          Show
          Kai Zheng added a comment - Updated patch with sync.
          Kai Zheng made changes -
          Description   Discussed with Natty about LdapGroupMapping, we need to improve it so that:
          1. It's possible to do different group mapping for different users/principals. For example, AD user should go to LdapGroupMapping service for group, but service principals such as hdfs, mapred can still use the default one ShellBasedUnixGroupsMapping;

          2. Multiple ADs can be supported to do LdapGroupMapping;

          3. It's possible to configure what kind of users/principals (regarding domain/realm is an option) should use which group mapping service/mechanism.
            Discussed with Natty about LdapGroupMapping, we need to improve it so that:
          1. It's possible to do different group mapping for different users/principals. For example, AD user should go to LdapGroupMapping service for group, but service principals such as hdfs, mapred can still use the default one ShellBasedUnixGroupsMapping;

          2. Multiple ADs can be supported to do LdapGroupMapping;

          3. It's possible to configure what kind of users/principals (regarding domain/realm is an option) should use which group mapping service/mechanism.

          4. It's possible to configure and combine multiple existing mapping providers without writing codes implementing new one.
          Kai Zheng made changes -
          Summary Enable to support multiple ADs and different group mapping for different user Support multiple group mapping providers
          Kai Zheng made changes -
          Attachment HADOOP-8943.patch [ 12553409 ]
          Hide
          Kai Zheng added a comment -

          Initial patch for review.

          Show
          Kai Zheng added a comment - Initial patch for review.
          Tianyou Li made changes -
          Description      Discussed with Natty about LdapGroupMapping, we need to improve it so that:
          1. It's possible to do different group mapping for different users/principals. For example, AD user should go to LdapGroupMapping service for group, but service principals such as hdfs, mapred can still use the default one ShellBasedUnixGroupsMapping;

          2. Multiple ADs can be supported to do LdapGroupMapping;

          3. It's possible to configure what kind of users/principals (regarding domain/realm is an option) should use which group mapping service/mechanism.
          Tianyou Li made changes -
          Field Original Value New Value
          Description Discussed with Natty about LdapGroupMapping, we need to improve it so that:
          1. It's possible to do different group mapping for different users/principals. For example, AD user should go to LdapGroupMapping service for group, but service principals such as hdfs, mapred can still use the default one ShellBasedUnixGroupsMapping;

          2. Multiple ADs can be supported to do LdapGroupMapping;

          3. It's possible to configure what kind of users/principals (regarding domain/realm is an option) should use which group mapping service/mechanism.
            
          Kai Zheng created issue -

            People

            • Assignee:
              Kai Zheng
              Reporter:
              Kai Zheng
            • Votes:
              1 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 504h
                504h
                Remaining:
                Remaining Estimate - 504h
                504h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development