Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13441

Document LdapGroupsMapping keystore password properties

    Details

      Description

      A few properties are not documented.
      hadoop.security.group.mapping.ldap.ssl.keystore.password
      This property is used as an alias to get password from credential providers, or, fall back to using the value as password in clear text. There is also a caveat that credential providers can not be a HDFS-based file system, as mentioned in HADOOP-11934, to prevent cyclic dependency issue.
      This should be documented in core-default.xml and GroupsMapping.md

      hadoop.security.credential.clear-text-fallback
      This property controls whether or not to fall back to storing credential password as cleartext.
      This should be documented in core-default.xml.

      hadoop.security.credential.provider.path
      This is mentioned in CredentialProvider API Guide, but not in core-default.xml

      The "Supported Features" in CredentialProvider API Guide should link back to GroupsMapping.md#LDAP Groups Mapping

      hadoop.security.credstore.java-keystore-provider.password-file
      This is the password file to protect credential files.

      1. HADOOP-13441.005.patch
        14 kB
        Yuanbo Liu
      2. HADOOP-13441.004.patch
        14 kB
        Yuanbo Liu
      3. HADOOP-13441.003.patch
        8 kB
        Yuanbo Liu
      4. HADOOP-13441.002.patch
        6 kB
        Yuanbo Liu
      5. HADOOP-13441.001.patch
        5 kB
        Yuanbo Liu

        Activity

        Hide
        yuanbo Yuanbo Liu added a comment -

        Wei-Chiu ChuangThank you for filing this, I'd like to work on this issue if you don't mind.

        Show
        yuanbo Yuanbo Liu added a comment - Wei-Chiu Chuang Thank you for filing this, I'd like to work on this issue if you don't mind.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Hi Yuanbo Liu sure no problem. Let me know when you have a patch ready for review. Thanks!

        Show
        jojochuang Wei-Chiu Chuang added a comment - Hi Yuanbo Liu sure no problem. Let me know when you have a patch ready for review. Thanks!
        Hide
        yuanbo Yuanbo Liu added a comment -

        Attach v1 patch for review. Wei-Chiu Chuang please help me review it if you have time. Thanks in advance !

        Show
        yuanbo Yuanbo Liu added a comment - Attach v1 patch for review. Wei-Chiu Chuang please help me review it if you have time. Thanks in advance !
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 32s trunk passed
        +1 compile 6m 43s trunk passed
        +1 mvnsite 0m 52s trunk passed
        +1 mvneclipse 0m 12s trunk passed
        +1 javadoc 0m 44s trunk passed
        +1 mvninstall 0m 37s the patch passed
        +1 compile 6m 40s the patch passed
        +1 javac 6m 40s the patch passed
        +1 mvnsite 0m 53s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 javadoc 0m 45s the patch passed
        -1 unit 7m 33s hadoop-common in the patch failed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        33m 3s



        Reason Tests
        Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics
          hadoop.conf.TestCommonConfigurationFields



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821270/HADOOP-13441.001.patch
        JIRA Issue HADOOP-13441
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml
        uname Linux 700097e0fb62 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 34ccaa8
        Default Java 1.8.0_101
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10137/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10137/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10137/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 32s trunk passed +1 compile 6m 43s trunk passed +1 mvnsite 0m 52s trunk passed +1 mvneclipse 0m 12s trunk passed +1 javadoc 0m 44s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 6m 40s the patch passed +1 javac 6m 40s the patch passed +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 javadoc 0m 45s the patch passed -1 unit 7m 33s hadoop-common in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 33m 3s Reason Tests Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics   hadoop.conf.TestCommonConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821270/HADOOP-13441.001.patch JIRA Issue HADOOP-13441 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml uname Linux 700097e0fb62 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 34ccaa8 Default Java 1.8.0_101 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10137/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10137/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10137/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        yuanbo Yuanbo Liu added a comment -

        uploaded v2 patch because of test failure.

        Show
        yuanbo Yuanbo Liu added a comment - uploaded v2 patch because of test failure.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 6m 37s trunk passed
        +1 compile 6m 46s trunk passed
        +1 checkstyle 0m 22s trunk passed
        -1 mvnsite 2m 10s hadoop-common in trunk failed.
        +1 mvneclipse 0m 12s trunk passed
        +1 findbugs 1m 33s trunk passed
        +1 javadoc 0m 43s trunk passed
        +1 mvninstall 0m 37s the patch passed
        +1 compile 6m 38s the patch passed
        +1 javac 6m 38s the patch passed
        +1 checkstyle 0m 23s the patch passed
        -1 mvnsite 2m 23s hadoop-common in the patch failed.
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 1m 40s the patch passed
        +1 javadoc 0m 45s the patch passed
        -1 unit 10m 0s hadoop-common in the patch failed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        43m 1s



        Reason Tests
        Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821277/HADOOP-13441.002.patch
        JIRA Issue HADOOP-13441
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
        uname Linux 60bb33329486 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 34ccaa8
        Default Java 1.8.0_101
        mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/10138/artifact/patchprocess/branch-mvnsite-hadoop-common-project_hadoop-common.txt
        findbugs v3.0.0
        mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/10138/artifact/patchprocess/patch-mvnsite-hadoop-common-project_hadoop-common.txt
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10138/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10138/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10138/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 37s trunk passed +1 compile 6m 46s trunk passed +1 checkstyle 0m 22s trunk passed -1 mvnsite 2m 10s hadoop-common in trunk failed. +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 33s trunk passed +1 javadoc 0m 43s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 6m 38s the patch passed +1 javac 6m 38s the patch passed +1 checkstyle 0m 23s the patch passed -1 mvnsite 2m 23s hadoop-common in the patch failed. +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 40s the patch passed +1 javadoc 0m 45s the patch passed -1 unit 10m 0s hadoop-common in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 43m 1s Reason Tests Failed junit tests hadoop.metrics2.impl.TestGangliaMetrics Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821277/HADOOP-13441.002.patch JIRA Issue HADOOP-13441 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 60bb33329486 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 34ccaa8 Default Java 1.8.0_101 mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/10138/artifact/patchprocess/branch-mvnsite-hadoop-common-project_hadoop-common.txt findbugs v3.0.0 mvnsite https://builds.apache.org/job/PreCommit-HADOOP-Build/10138/artifact/patchprocess/patch-mvnsite-hadoop-common-project_hadoop-common.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10138/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10138/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10138/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        yuanbo Yuanbo Liu added a comment -

        The test failure seems related to HADOOP-12588 and HADOOP-13439

        Show
        yuanbo Yuanbo Liu added a comment - The test failure seems related to HADOOP-12588 and HADOOP-13439
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Hi Yuanbo Liu, first of all, thanks for contributing the patch. Let's work together to document these properties better!

        • hadoop.security.group.mapping.ldap.ssl.keystore.password.file, I think it would be more accurate to state that the path must point to a local file.
        • hadoop.security.group.mapping.ldap.ssl.keystore.password
          The code actually works like this: if credential providers are configured, this property is used as an alias to get the password from credential providers. If the alias can not be found and if hadoop.security.credential.clear-text-fallback is true, LDAPGroupsMapping uses the value of this property for password. If the value is not configured, LDAPGroupsMapping reads password from the file in hadoop.security.group.mapping.ldap.ssl.keystore.password.file.
        • hadoop.security.credential.clear-text-fallback
          I think it may be more precise to state that when using an alias to find a credential entry, if it is not found, whether or not to fallback and to use the alias as the configuration property key and return its value.
        • hadoop.security.group.mapping.ldap.bind.password
          I missed this property in the beginning, but
          the story here is similar to hadoop.security.group.mapping.ldap.ssl.keystore.password, except this one is for authentication password with LDAP server.
          if credential providers are configured, this property is used as an alias to get the password from credential providers. If the alias can not be found and if hadoop.security.credential.clear-text-fallback is true, LDAPGroupsMapping uses the value of this property for password. If the value is not configured, LDAPGroupsMapping reads password from the file in hadoop.security.group.mapping.ldap.bind.password.file.
        Show
        jojochuang Wei-Chiu Chuang added a comment - Hi Yuanbo Liu , first of all, thanks for contributing the patch. Let's work together to document these properties better! hadoop.security.group.mapping.ldap.ssl.keystore.password.file , I think it would be more accurate to state that the path must point to a local file. hadoop.security.group.mapping.ldap.ssl.keystore.password The code actually works like this: if credential providers are configured, this property is used as an alias to get the password from credential providers. If the alias can not be found and if hadoop.security.credential.clear-text-fallback is true, LDAPGroupsMapping uses the value of this property for password. If the value is not configured, LDAPGroupsMapping reads password from the file in hadoop.security.group.mapping.ldap.ssl.keystore.password.file . hadoop.security.credential.clear-text-fallback I think it may be more precise to state that when using an alias to find a credential entry, if it is not found, whether or not to fallback and to use the alias as the configuration property key and return its value. hadoop.security.group.mapping.ldap.bind.password I missed this property in the beginning, but the story here is similar to hadoop.security.group.mapping.ldap.ssl.keystore.password , except this one is for authentication password with LDAP server. if credential providers are configured, this property is used as an alias to get the password from credential providers. If the alias can not be found and if hadoop.security.credential.clear-text-fallback is true, LDAPGroupsMapping uses the value of this property for password. If the value is not configured, LDAPGroupsMapping reads password from the file in hadoop.security.group.mapping.ldap.bind.password.file .
        Hide
        yuanbo Yuanbo Liu added a comment -

        Wei-Chiu Chuang Thanks for your comments!

        this property is used as an alias to get the password from credential providers. If the alias can not be found and if

        I change a bit and use this sentence below:

          this property name is used as an alias to get the password from credential providers. If the password can not be found. 

        LDAPGroupsMapping use the property name as an alias, then get credential entry from credential providers, finally get the password from the credential entry. these are the code process details and I describe it as "get the password from credential providers" to make the description compact and uniform.
        I've uploaded v3 patch to address your comments. Hope to get your thoughts, thanks again for your time!

        Show
        yuanbo Yuanbo Liu added a comment - Wei-Chiu Chuang Thanks for your comments! this property is used as an alias to get the password from credential providers. If the alias can not be found and if I change a bit and use this sentence below: this property name is used as an alias to get the password from credential providers. If the password can not be found. LDAPGroupsMapping use the property name as an alias, then get credential entry from credential providers, finally get the password from the credential entry. these are the code process details and I describe it as "get the password from credential providers" to make the description compact and uniform. I've uploaded v3 patch to address your comments. Hope to get your thoughts, thanks again for your time!
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        +1 mvninstall 8m 44s trunk passed
        +1 compile 8m 18s trunk passed
        +1 checkstyle 0m 28s trunk passed
        +1 mvnsite 1m 3s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        +1 findbugs 1m 36s trunk passed
        +1 javadoc 0m 52s trunk passed
        +1 mvninstall 0m 44s the patch passed
        +1 compile 8m 9s the patch passed
        +1 javac 8m 9s the patch passed
        +1 checkstyle 0m 28s the patch passed
        +1 mvnsite 1m 5s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 findbugs 1m 43s the patch passed
        +1 javadoc 0m 49s the patch passed
        +1 unit 9m 11s hadoop-common in the patch passed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        45m 50s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821998/HADOOP-13441.003.patch
        JIRA Issue HADOOP-13441
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle
        uname Linux 9e1dffe80496 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8f1c374
        Default Java 1.8.0_101
        findbugs v3.0.0
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10173/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10173/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 44s trunk passed +1 compile 8m 18s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 3s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 36s trunk passed +1 javadoc 0m 52s trunk passed +1 mvninstall 0m 44s the patch passed +1 compile 8m 9s the patch passed +1 javac 8m 9s the patch passed +1 checkstyle 0m 28s the patch passed +1 mvnsite 1m 5s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 1m 43s the patch passed +1 javadoc 0m 49s the patch passed +1 unit 9m 11s hadoop-common in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 45m 50s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12821998/HADOOP-13441.003.patch JIRA Issue HADOOP-13441 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit xml findbugs checkstyle uname Linux 9e1dffe80496 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8f1c374 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10173/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10173/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Hi Yuanbo Liu thanks again for the updated patch. This is largely good.

        Couple of comments:

        • instead of skipping the properties in TestCommonConfigurationFields, can you define these property constants in CommonConfigurationKeysPublic, for example,
          CommonConfigurationKeysPublic.java
          public static final String  HADOOP_SECURITY_CREDENTIAL_PASSWORD_FILE_KEY = "hadoop.security.credstore.java-keystore-provider.password-file";
          

          and then in AbstractJavaKeyStoreProvider.java:

          AbstractJavaKeyStoreProvider.java
          public static final String CREDENTIAL_PASSWORD_FILE_KEY = CommonConfigurationKeysPublic.HADOOP_SECURITY_CREDENTIAL_PASSWORD_FILE_KEY;
          
        • hadoop.security.group.mapping.ldap.bind.password.file

          + The path to a file containing the password of the bind user. If
          + the password is not configured in credential providers and the property
          + hadoop.security.group.mapping.ldap.bind.password, LDAPGroupsMapping
          + reads password from the file.

          should be "and the property hadoop.security.group.mapping.ldap.bind.password is not set"

        Similarly the same change is needed for hadoop.security.group.mapping.ldap.ssl.keystore.password.file.

        • GroupsMapping.md

          +In addition, specify the path to the keystore file for SSL connection in `hadoop.security.group.mapping.ldap.ssl.keystore` and keystore password in `hadoop.security.group.mapping.ldap.ssl.keystore.password`, at the same time, make sure `hadoop.security.credential.clear-text-fallback` is true.
          +Alternatively, store the keystore password in a file, and point `hadoop.security.group.mapping.ldap.ssl.keystore.password.file` to that file.
          +For security purposes, this file should be readable only by the Unix user running the daemons, and for preventing recursive dependency, this file should be a local file.

          This is good. Can you also add that "keystore password in `hadoop.security.group.mapping.ldap.ssl.keystore.password`" is highly discouraged, because it exposes the password in the configuration file. Instead, use the credential file and use `hadoop.security.group.mapping.ldap.ssl.keystore.password` as the alias in the credential file for password, or use `hadoop.security.group.mapping.ldap.ssl.keystore.password.file`.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Hi Yuanbo Liu thanks again for the updated patch. This is largely good. Couple of comments: instead of skipping the properties in TestCommonConfigurationFields, can you define these property constants in CommonConfigurationKeysPublic , for example, CommonConfigurationKeysPublic.java public static final String HADOOP_SECURITY_CREDENTIAL_PASSWORD_FILE_KEY = "hadoop.security.credstore.java-keystore-provider.password-file" ; and then in AbstractJavaKeyStoreProvider.java: AbstractJavaKeyStoreProvider.java public static final String CREDENTIAL_PASSWORD_FILE_KEY = CommonConfigurationKeysPublic.HADOOP_SECURITY_CREDENTIAL_PASSWORD_FILE_KEY; hadoop.security.group.mapping.ldap.bind.password.file + The path to a file containing the password of the bind user. If + the password is not configured in credential providers and the property + hadoop.security.group.mapping.ldap.bind.password, LDAPGroupsMapping + reads password from the file. should be "and the property hadoop.security.group.mapping.ldap.bind.password is not set" Similarly the same change is needed for hadoop.security.group.mapping.ldap.ssl.keystore.password.file . GroupsMapping.md +In addition, specify the path to the keystore file for SSL connection in `hadoop.security.group.mapping.ldap.ssl.keystore` and keystore password in `hadoop.security.group.mapping.ldap.ssl.keystore.password`, at the same time, make sure `hadoop.security.credential.clear-text-fallback` is true. +Alternatively, store the keystore password in a file, and point `hadoop.security.group.mapping.ldap.ssl.keystore.password.file` to that file. +For security purposes, this file should be readable only by the Unix user running the daemons, and for preventing recursive dependency, this file should be a local file. This is good. Can you also add that "keystore password in `hadoop.security.group.mapping.ldap.ssl.keystore.password`" is highly discouraged, because it exposes the password in the configuration file. Instead, use the credential file and use `hadoop.security.group.mapping.ldap.ssl.keystore.password` as the alias in the credential file for password, or use `hadoop.security.group.mapping.ldap.ssl.keystore.password.file`.
        Hide
        yuanbo Yuanbo Liu added a comment -

        Wei-Chiu Chuang Thanks for your detail comments!
        Sorry for the late response. I will prepare a new patch soon.

        Show
        yuanbo Yuanbo Liu added a comment - Wei-Chiu Chuang Thanks for your detail comments! Sorry for the late response. I will prepare a new patch soon.
        Hide
        yuanbo Yuanbo Liu added a comment -

        This is good. Can you also add that "keystore password in `hadoop.security.group.mapping.ldap.ssl.keystore.password`" is highly discouraged......

        This description overlaps the approach description in GroupsMapping.md. So I just add a new short line to address it.

        The second approach aka using `hadoop.security.group.mapping.ldap.ssl.keystore.password` is highly discouraged because it exposes the password in the configuration file.

        Others look good to me, upload v4 patch. Wei-Chiu Chuang Thanks a lot for your time!

        Show
        yuanbo Yuanbo Liu added a comment - This is good. Can you also add that "keystore password in `hadoop.security.group.mapping.ldap.ssl.keystore.password`" is highly discouraged...... This description overlaps the approach description in GroupsMapping.md. So I just add a new short line to address it. The second approach aka using `hadoop.security.group.mapping.ldap.ssl.keystore.password` is highly discouraged because it exposes the password in the configuration file. Others look good to me, upload v4 patch. Wei-Chiu Chuang Thanks a lot for your time!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 6m 34s trunk passed
        +1 compile 6m 48s trunk passed
        +1 checkstyle 0m 26s trunk passed
        +1 mvnsite 0m 53s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 17s trunk passed
        +1 javadoc 0m 44s trunk passed
        +1 mvninstall 0m 37s the patch passed
        +1 compile 6m 39s the patch passed
        +1 javac 6m 39s the patch passed
        +1 checkstyle 0m 27s the patch passed
        +1 mvnsite 0m 51s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 findbugs 1m 26s the patch passed
        +1 javadoc 0m 45s the patch passed
        -1 unit 7m 36s hadoop-common in the patch failed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        37m 30s



        Reason Tests
        Failed junit tests hadoop.net.TestDNS



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822949/HADOOP-13441.004.patch
        JIRA Issue HADOOP-13441
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 983558f1a60c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d00d3ad
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10217/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10217/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10217/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 34s trunk passed +1 compile 6m 48s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 53s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 17s trunk passed +1 javadoc 0m 44s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 6m 39s the patch passed +1 javac 6m 39s the patch passed +1 checkstyle 0m 27s the patch passed +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 1m 26s the patch passed +1 javadoc 0m 45s the patch passed -1 unit 7m 36s hadoop-common in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 37m 30s Reason Tests Failed junit tests hadoop.net.TestDNS Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12822949/HADOOP-13441.004.patch JIRA Issue HADOOP-13441 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 983558f1a60c 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d00d3ad Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10217/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10217/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10217/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Hello Yuanbo Liu thanks again for the patch. It looks almost done!

        There is just one thing and I apologize for the confusion:

        +The second approach aka using `hadoop.security.group.mapping.ldap.ssl.keystore.password` is highly discouraged because it exposes the password in the configuration file.

        I think you meant to say "the first approach"?

        +1 after fixing this.

        Show
        jojochuang Wei-Chiu Chuang added a comment - Hello Yuanbo Liu thanks again for the patch. It looks almost done! There is just one thing and I apologize for the confusion: +The second approach aka using `hadoop.security.group.mapping.ldap.ssl.keystore.password` is highly discouraged because it exposes the password in the configuration file. I think you meant to say "the first approach"? +1 after fixing this.
        Hide
        yuanbo Yuanbo Liu added a comment -

        Wei-Chiu Chuang Sorry about the mistake, that's my fault.
        Uploaded v5 patch to address your comment. Thanks very much!

        Show
        yuanbo Yuanbo Liu added a comment - Wei-Chiu Chuang Sorry about the mistake, that's my fault. Uploaded v5 patch to address your comment. Thanks very much!
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 14s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 16s trunk passed
        +1 compile 6m 47s trunk passed
        +1 checkstyle 0m 26s trunk passed
        +1 mvnsite 0m 54s trunk passed
        +1 mvneclipse 0m 13s trunk passed
        +1 findbugs 1m 19s trunk passed
        +1 javadoc 0m 44s trunk passed
        +1 mvninstall 0m 37s the patch passed
        +1 compile 6m 49s the patch passed
        +1 javac 6m 49s the patch passed
        +1 checkstyle 0m 27s the patch passed
        +1 mvnsite 0m 54s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 1m 33s the patch passed
        +1 javadoc 0m 45s the patch passed
        -1 unit 7m 43s hadoop-common in the patch failed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        38m 42s



        Reason Tests
        Failed junit tests hadoop.security.ssl.TestSSLFactory



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823165/HADOOP-13441.005.patch
        JIRA Issue HADOOP-13441
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 7f0af75dcfa0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / aea3e65
        Default Java 1.8.0_101
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10223/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt
        Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10223/testReport/
        modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
        Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10223/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 16s trunk passed +1 compile 6m 47s trunk passed +1 checkstyle 0m 26s trunk passed +1 mvnsite 0m 54s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 19s trunk passed +1 javadoc 0m 44s trunk passed +1 mvninstall 0m 37s the patch passed +1 compile 6m 49s the patch passed +1 javac 6m 49s the patch passed +1 checkstyle 0m 27s the patch passed +1 mvnsite 0m 54s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 1m 33s the patch passed +1 javadoc 0m 45s the patch passed -1 unit 7m 43s hadoop-common in the patch failed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 38m 42s Reason Tests Failed junit tests hadoop.security.ssl.TestSSLFactory Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12823165/HADOOP-13441.005.patch JIRA Issue HADOOP-13441 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 7f0af75dcfa0 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aea3e65 Default Java 1.8.0_101 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/10223/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common.txt Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/10223/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/10223/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        +1. The test failure is unrelated. I'll file a new jira to fix the failed test if it's not filed already.

        Show
        jojochuang Wei-Chiu Chuang added a comment - +1. The test failure is unrelated. I'll file a new jira to fix the failed test if it's not filed already.
        Hide
        jojochuang Wei-Chiu Chuang added a comment -

        Committed it to trunk and branch-2. Thanks Yuanbo Liu for contributing the patch!

        Show
        jojochuang Wei-Chiu Chuang added a comment - Committed it to trunk and branch-2. Thanks Yuanbo Liu for contributing the patch!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #10263 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10263/)
        HADOOP-13441. Document LdapGroupsMapping keystore password properties. (weichiu: rev d892ae9576d07d01927443b6dc6c934a6c2f317f)

        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialProviderFactory.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialProvider.java
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java
        • hadoop-common-project/hadoop-common/src/site/markdown/CredentialProviderAPI.md
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java
        • hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
        • hadoop-common-project/hadoop-common/src/site/markdown/GroupsMapping.md
        • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10263 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10263/ ) HADOOP-13441 . Document LdapGroupsMapping keystore password properties. (weichiu: rev d892ae9576d07d01927443b6dc6c934a6c2f317f) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialProviderFactory.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/CredentialProvider.java hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/CommonConfigurationKeysPublic.java hadoop-common-project/hadoop-common/src/site/markdown/CredentialProviderAPI.md hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java hadoop-common-project/hadoop-common/src/main/resources/core-default.xml hadoop-common-project/hadoop-common/src/site/markdown/GroupsMapping.md hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/AbstractJavaKeyStoreProvider.java
        Hide
        yuanbo Yuanbo Liu added a comment -

        Thanks very much Wei-Chiu Chuang

        Show
        yuanbo Yuanbo Liu added a comment - Thanks very much Wei-Chiu Chuang

          People

          • Assignee:
            yuanbo Yuanbo Liu
            Reporter:
            jojochuang Wei-Chiu Chuang
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development