Hadoop Common
  1. Hadoop Common
  2. HADOOP-8712

Change default hadoop.security.group.mapping

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.0.2-alpha
    • Fix Version/s: 2.0.3-alpha
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      The default group mapping policy has been changed to JniBasedUnixGroupsNetgroupMappingWithFallback. This should maintain the same semantics as the prior default for most users.

      Description

      Change the hadoop.security.group.mapping in core-site to JniBasedUnixGroupsNetgroupMappingWithFallback

      1. HADOOP-8712-v1.patch
        2 kB
        Robert Parker
      2. HADOOP-8712-v2.patch
        3 kB
        Robert Parker

        Issue Links

          Activity

          Hide
          Jason Lowe added a comment -

          This change broke the start-dfs.sh script for me because hdfs getconf now prints a warning when the native libraries are not present and the log text is misinterpreted as a machine name, see HDFS-4427.

          Show
          Jason Lowe added a comment - This change broke the start-dfs.sh script for me because hdfs getconf now prints a warning when the native libraries are not present and the log text is misinterpreted as a machine name, see HDFS-4427 .
          Hide
          Chris Nauroth added a comment -

          This change breaks on Windows (branch-trunk-win) due to lack of a Windows implementation for the native method in hadoop.dll. The fallback logic is a one-time check to see if hadoop.dll loaded successfully, so with this kind of failure, it won't fall back to ShellBasedUnixGroupsMapping. I've filed HADOOP-9232 to track it. Meanwhile, a workaround on Windows is to set the configuration back to ShellBasedUnixGroupsMapping manually in core-site.xml:

          <property>
            <name>hadoop.security.group.mapping</name>
            <value>org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback</value>
          </property>
          
          Show
          Chris Nauroth added a comment - This change breaks on Windows (branch-trunk-win) due to lack of a Windows implementation for the native method in hadoop.dll. The fallback logic is a one-time check to see if hadoop.dll loaded successfully, so with this kind of failure, it won't fall back to ShellBasedUnixGroupsMapping . I've filed HADOOP-9232 to track it. Meanwhile, a workaround on Windows is to set the configuration back to ShellBasedUnixGroupsMapping manually in core-site.xml: <property> <name>hadoop.security.group.mapping</name> <value>org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback</value> </property>
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1315 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1315/)
          HADOOP-8712. Change default hadoop.security.group.mapping to JniBasedUnixGroupsNetgroupMappingWithFallback. Contributed by Robert Parker. (Revision 1433624)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1433624
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/hdfs_permissions_guide.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1315 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1315/ ) HADOOP-8712 . Change default hadoop.security.group.mapping to JniBasedUnixGroupsNetgroupMappingWithFallback. Contributed by Robert Parker. (Revision 1433624) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1433624 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/hdfs_permissions_guide.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1287 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1287/)
          HADOOP-8712. Change default hadoop.security.group.mapping to JniBasedUnixGroupsNetgroupMappingWithFallback. Contributed by Robert Parker. (Revision 1433624)

          Result = FAILURE
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1433624
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/hdfs_permissions_guide.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1287 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1287/ ) HADOOP-8712 . Change default hadoop.security.group.mapping to JniBasedUnixGroupsNetgroupMappingWithFallback. Contributed by Robert Parker. (Revision 1433624) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1433624 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/hdfs_permissions_guide.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #98 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/98/)
          HADOOP-8712. Change default hadoop.security.group.mapping to JniBasedUnixGroupsNetgroupMappingWithFallback. Contributed by Robert Parker. (Revision 1433624)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1433624
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/hdfs_permissions_guide.xml
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #98 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/98/ ) HADOOP-8712 . Change default hadoop.security.group.mapping to JniBasedUnixGroupsNetgroupMappingWithFallback. Contributed by Robert Parker. (Revision 1433624) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1433624 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/hdfs_permissions_guide.xml
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3244 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3244/)
          HADOOP-8712. Change default hadoop.security.group.mapping to JniBasedUnixGroupsNetgroupMappingWithFallback. Contributed by Robert Parker. (Revision 1433624)

          Result = SUCCESS
          todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1433624
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/hdfs_permissions_guide.xml
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3244 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3244/ ) HADOOP-8712 . Change default hadoop.security.group.mapping to JniBasedUnixGroupsNetgroupMappingWithFallback. Contributed by Robert Parker. (Revision 1433624) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1433624 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/resources/core-default.xml /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/docs/src/documentation/content/xdocs/hdfs_permissions_guide.xml
          Hide
          Todd Lipcon added a comment -

          Committed to branch-2 and trunk. Thanks, Robert.

          Show
          Todd Lipcon added a comment - Committed to branch-2 and trunk. Thanks, Robert.
          Hide
          Eli Collins added a comment -

          +1 from me, I thought this had gone in already.

          Show
          Eli Collins added a comment - +1 from me, I thought this had gone in already.
          Hide
          Todd Lipcon added a comment -

          Anything holding this up? Looks ready to go as of late August. I'll commit it based on the earlier +1s unless I hear any objections.

          Show
          Todd Lipcon added a comment - Anything holding this up? Looks ready to go as of late August. I'll commit it based on the earlier +1s unless I hear any objections.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12542926/HADOOP-8712-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in .

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1379//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1379//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542926/HADOOP-8712-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1379//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1379//console This message is automatically generated.
          Hide
          Robert Parker added a comment -

          Corrected spelling error, explicitly stated the fallback mechanism, moved the description to core-default.xml with a reference in the hdfs-permission-guide.xml to eliminate multiple maintenance points.

          Show
          Robert Parker added a comment - Corrected spelling error, explicitly stated the fallback mechanism, moved the description to core-default.xml with a reference in the hdfs-permission-guide.xml to eliminate multiple maintenance points.
          Hide
          Harsh J added a comment -

          One more nit: Lets add the wide description to the core-default.xml as well?

          Show
          Harsh J added a comment - One more nit: Lets add the wide description to the core-default.xml as well?
          Hide
          Harsh J added a comment -

          +1 overall

          Just some description nits:

          • s/resovle/resolve
          • The description doesn't mention that the groups fallback is used only when JNI is unavailable. It is implicit currently, lets make it explicit.
          Show
          Harsh J added a comment - +1 overall Just some description nits: s/resovle/resolve The description doesn't mention that the groups fallback is used only when JNI is unavailable. It is implicit currently, lets make it explicit.
          Hide
          Robert Parker added a comment -

          HDFS-3837 and HDFS-3850 addresses the findbugs issues

          Show
          Robert Parker added a comment - HDFS-3837 and HDFS-3850 addresses the findbugs issues
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12542136/HADOOP-8712-v1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.ha.TestZKFailoverController
          org.apache.hadoop.hdfs.TestReplication

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1352//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/1352//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1352//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542136/HADOOP-8712-v1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.ha.TestZKFailoverController org.apache.hadoop.hdfs.TestReplication +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1352//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/1352//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1352//console This message is automatically generated.
          Hide
          Robert Parker added a comment -

          Changed the default to JniBasedUnixGroupsMappingWithFallback. Updated the documentation.

          Show
          Robert Parker added a comment - Changed the default to JniBasedUnixGroupsMappingWithFallback. Updated the documentation.
          Hide
          Robert Joseph Evans added a comment -

          I think it should be JniBasedUnixGroupsMappingWithFallback, because it falls back to ShellBasedUnixGroupsMapping which is the current default.

          Show
          Robert Joseph Evans added a comment - I think it should be JniBasedUnixGroupsMappingWithFallback, because it falls back to ShellBasedUnixGroupsMapping which is the current default.

            People

            • Assignee:
              Robert Parker
              Reporter:
              Robert Parker
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development