Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Add a NullGroupMapping for cases where user groups are not used. ShellBasedUnixGroupMapping can be used in places where latency is not important. In places like starting a container, it's worth in to avoid the extra fork and exec.

      1. HADOOP-12566.001.patch
        4 kB
        Daniel Templeton
      2. HADOOP-12566.002.patch
        4 kB
        Daniel Templeton

        Issue Links

          Activity

          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 5s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 3m 6s trunk passed
          +1 compile 4m 14s trunk passed with JDK v1.8.0_60
          +1 compile 4m 6s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 14s trunk passed
          +1 mvnsite 1m 1s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 1m 37s trunk passed
          +1 javadoc 0m 51s trunk passed with JDK v1.8.0_60
          +1 javadoc 1m 2s trunk passed with JDK v1.7.0_79
          +1 mvninstall 1m 42s the patch passed
          +1 compile 4m 15s the patch passed with JDK v1.8.0_60
          -1 javac 9m 28s root-jdk1.8.0_60 with JDK v1.8.0_60 generated 2 new issues (was 145, now 147).
          +1 javac 4m 15s the patch passed
          +1 compile 4m 10s the patch passed with JDK v1.7.0_79
          -1 javac 13m 38s root-jdk1.7.0_79 with JDK v1.7.0_79 generated 2 new issues (was 147, now 149).
          +1 javac 4m 10s the patch passed
          +1 checkstyle 0m 13s the patch passed
          +1 mvnsite 0m 53s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 1m 49s the patch passed
          +1 javadoc 0m 54s the patch passed with JDK v1.8.0_60
          +1 javadoc 1m 3s the patch passed with JDK v1.7.0_79
          -1 unit 16m 0s hadoop-common in the patch failed with JDK v1.8.0_60.
          +1 unit 9m 8s hadoop-common in the patch passed with JDK v1.7.0_79.
          +1 asflicense 0m 25s Patch does not generate ASF License warnings.
          58m 16s



          Reason Tests
          JDK v1.8.0_60 Timed out junit tests org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-15
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772410/HADOOP-12566.001.patch
          JIRA Issue HADOOP-12566
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 8670b1341bd7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh
          git revision trunk / 855d529
          findbugs v3.0.0
          javac root-jdk1.8.0_60: https://builds.apache.org/job/PreCommit-HADOOP-Build/8073/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_60.txt
          javac root-jdk1.7.0_79: https://builds.apache.org/job/PreCommit-HADOOP-Build/8073/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_79.txt
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8073/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8073/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8073/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 226MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8073/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 5s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 3m 6s trunk passed +1 compile 4m 14s trunk passed with JDK v1.8.0_60 +1 compile 4m 6s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 14s trunk passed +1 mvnsite 1m 1s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 1m 37s trunk passed +1 javadoc 0m 51s trunk passed with JDK v1.8.0_60 +1 javadoc 1m 2s trunk passed with JDK v1.7.0_79 +1 mvninstall 1m 42s the patch passed +1 compile 4m 15s the patch passed with JDK v1.8.0_60 -1 javac 9m 28s root-jdk1.8.0_60 with JDK v1.8.0_60 generated 2 new issues (was 145, now 147). +1 javac 4m 15s the patch passed +1 compile 4m 10s the patch passed with JDK v1.7.0_79 -1 javac 13m 38s root-jdk1.7.0_79 with JDK v1.7.0_79 generated 2 new issues (was 147, now 149). +1 javac 4m 10s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 53s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 49s the patch passed +1 javadoc 0m 54s the patch passed with JDK v1.8.0_60 +1 javadoc 1m 3s the patch passed with JDK v1.7.0_79 -1 unit 16m 0s hadoop-common in the patch failed with JDK v1.8.0_60. +1 unit 9m 8s hadoop-common in the patch passed with JDK v1.7.0_79. +1 asflicense 0m 25s Patch does not generate ASF License warnings. 58m 16s Reason Tests JDK v1.8.0_60 Timed out junit tests org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-15 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772410/HADOOP-12566.001.patch JIRA Issue HADOOP-12566 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 8670b1341bd7 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh git revision trunk / 855d529 findbugs v3.0.0 javac root-jdk1.8.0_60: https://builds.apache.org/job/PreCommit-HADOOP-Build/8073/artifact/patchprocess/diff-compile-javac-root-jdk1.8.0_60.txt javac root-jdk1.7.0_79: https://builds.apache.org/job/PreCommit-HADOOP-Build/8073/artifact/patchprocess/diff-compile-javac-root-jdk1.7.0_79.txt unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8073/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8073/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_60.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8073/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 226MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8073/console This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Daniel Temme thank you for taking this issue. I've not looked at the patch deeper, but could you fix javac warnings?

          [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestNullGroupsMapping.java: /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestNullGroupsMapping.java uses unchecked or unsafe operations.
          [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestNullGroupsMapping.java: Recompile with -Xlint:unchecked for details.

          Show
          ozawa Tsuyoshi Ozawa added a comment - Daniel Temme thank you for taking this issue. I've not looked at the patch deeper, but could you fix javac warnings? [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestNullGroupsMapping.java: /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestNullGroupsMapping.java uses unchecked or unsafe operations. [WARNING] /testptch/hadoop/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestNullGroupsMapping.java: Recompile with -Xlint:unchecked for details.
          Hide
          templedf Daniel Templeton added a comment -

          Thanks, [ozawa]. For some reason I'm not seeing the unchecked compile errors when I build locally with Maven, but I was able to see them running javac by hand.

          Here's a patch with the issue resolved.

          Show
          templedf Daniel Templeton added a comment - Thanks, [ozawa] . For some reason I'm not seeing the unchecked compile errors when I build locally with Maven, but I was able to see them running javac by hand. Here's a patch with the issue resolved.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s docker + precommit patch detected.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 4m 10s trunk passed
          +1 compile 6m 4s trunk passed with JDK v1.8.0_60
          +1 compile 5m 32s trunk passed with JDK v1.7.0_79
          +1 checkstyle 0m 21s trunk passed
          +1 mvnsite 1m 7s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 1m 53s trunk passed
          +1 javadoc 1m 1s trunk passed with JDK v1.8.0_60
          +1 javadoc 1m 13s trunk passed with JDK v1.7.0_79
          +1 mvninstall 1m 37s the patch passed
          +1 compile 5m 1s the patch passed with JDK v1.8.0_60
          +1 javac 5m 1s the patch passed
          +1 compile 4m 49s the patch passed with JDK v1.7.0_79
          +1 javac 4m 49s the patch passed
          +1 checkstyle 0m 16s the patch passed
          +1 mvnsite 0m 58s the patch passed
          +1 mvneclipse 0m 16s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 2m 3s the patch passed
          +1 javadoc 1m 2s the patch passed with JDK v1.8.0_60
          +1 javadoc 1m 10s the patch passed with JDK v1.7.0_79
          +1 unit 7m 56s hadoop-common in the patch passed with JDK v1.8.0_60.
          -1 unit 7m 52s hadoop-common in the patch failed with JDK v1.7.0_79.
          +1 asflicense 0m 27s Patch does not generate ASF License warnings.
          56m 29s



          Reason Tests
          JDK v1.7.0_79 Failed junit tests hadoop.ipc.TestDecayRpcScheduler



          Subsystem Report/Notes
          Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-16
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772590/HADOOP-12566.002.patch
          JIRA Issue HADOOP-12566
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 26647ee2f31b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh
          git revision trunk / 02653ad
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8074/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt
          unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8074/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt
          JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8074/testReport/
          modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common
          Max memory used 228MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8074/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s docker + precommit patch detected. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 4m 10s trunk passed +1 compile 6m 4s trunk passed with JDK v1.8.0_60 +1 compile 5m 32s trunk passed with JDK v1.7.0_79 +1 checkstyle 0m 21s trunk passed +1 mvnsite 1m 7s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 1m 53s trunk passed +1 javadoc 1m 1s trunk passed with JDK v1.8.0_60 +1 javadoc 1m 13s trunk passed with JDK v1.7.0_79 +1 mvninstall 1m 37s the patch passed +1 compile 5m 1s the patch passed with JDK v1.8.0_60 +1 javac 5m 1s the patch passed +1 compile 4m 49s the patch passed with JDK v1.7.0_79 +1 javac 4m 49s the patch passed +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 58s the patch passed +1 mvneclipse 0m 16s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 2m 3s the patch passed +1 javadoc 1m 2s the patch passed with JDK v1.8.0_60 +1 javadoc 1m 10s the patch passed with JDK v1.7.0_79 +1 unit 7m 56s hadoop-common in the patch passed with JDK v1.8.0_60. -1 unit 7m 52s hadoop-common in the patch failed with JDK v1.7.0_79. +1 asflicense 0m 27s Patch does not generate ASF License warnings. 56m 29s Reason Tests JDK v1.7.0_79 Failed junit tests hadoop.ipc.TestDecayRpcScheduler Subsystem Report/Notes Docker Client=1.7.1 Server=1.7.1 Image:test-patch-base-hadoop-date2015-11-16 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12772590/HADOOP-12566.002.patch JIRA Issue HADOOP-12566 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 26647ee2f31b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/patchprocess/apache-yetus-fa12328/precommit/personality/hadoop.sh git revision trunk / 02653ad findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-HADOOP-Build/8074/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt unit test logs https://builds.apache.org/job/PreCommit-HADOOP-Build/8074/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_79.txt JDK v1.7.0_79 Test Results https://builds.apache.org/job/PreCommit-HADOOP-Build/8074/testReport/ modules C: hadoop-common-project/hadoop-common U: hadoop-common-project/hadoop-common Max memory used 228MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-HADOOP-Build/8074/console This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Unit test failure is unrelated.

          Show
          templedf Daniel Templeton added a comment - Unit test failure is unrelated.
          Hide
          kasha Karthik Kambatla added a comment -

          Fairly straight-forward patch. +1. Committing this..

          Show
          kasha Karthik Kambatla added a comment - Fairly straight-forward patch. +1. Committing this..
          Hide
          kasha Karthik Kambatla added a comment -

          Just committed this to trunk and branch-2.

          Show
          kasha Karthik Kambatla added a comment - Just committed this to trunk and branch-2.
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks for filing and working on this, Daniel Templeton

          Show
          kasha Karthik Kambatla added a comment - Thanks for filing and working on this, Daniel Templeton
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9020 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9020/)
          HADOOP-12566. Add NullGroupMapping. (Daniel Templeton via kasha) (kasha: rev 8b1837fcd1f01d881deacd72b4e548ec2bdcb7ad)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/NullGroupsMapping.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestNullGroupsMapping.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9020 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9020/ ) HADOOP-12566 . Add NullGroupMapping. (Daniel Templeton via kasha) (kasha: rev 8b1837fcd1f01d881deacd72b4e548ec2bdcb7ad) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/NullGroupsMapping.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestNullGroupsMapping.java

            People

            • Assignee:
              templedf Daniel Templeton
              Reporter:
              templedf Daniel Templeton
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development