Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2457

job submission should inject group.name (on the JT side)

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 0.21.0, 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Until Hadoop 0.20, the JobClient was injecting the property 'group.name' on the JobConf submitted to the JobTracker.

      Since Hadoop 0.21, due to security related changes, this is not done anymore.

      This breaks backwards compatibility for jobs/components that expect the 'group.name' to be automatically set at submission time.

      An example of a component being affected by this change is the FairScheduler where it is common to use the group.name as pool name. Different from other properties, a special characteristic of the group.name is that its value cannot be tampered by a user.

      For security reasons this should not be done (as it was done before) in the JobClient side. Instead, it should be done in the JobTracker when the JobConf is received.

      1. MAPREDUCE-2457-1.patch
        2 kB
        Alejandro Abdelnur
      2. MAPREDUCE-2457.patch
        2 kB
        Alejandro Abdelnur

        Activity

        Hide
        Alejandro Abdelnur added a comment -

        testcase verifies that the primary group of the user submitting the job is used as pool name. This is done in on of the FairScheduler testcases.

        Note that the group.name will not be avail to tasks as the JT does not re-write the job.xml to HDFS

        Show
        Alejandro Abdelnur added a comment - testcase verifies that the primary group of the user submitting the job is used as pool name. This is done in on of the FairScheduler testcases. Note that the group.name will not be avail to tasks as the JT does not re-write the job.xml to HDFS
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12477708/MAPREDUCE-2457.patch
        against trunk revision 1097345.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/195//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12477708/MAPREDUCE-2457.patch against trunk revision 1097345. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/195//console This message is automatically generated.
        Hide
        Alejandro Abdelnur added a comment -

        patch works against 22, needs to work against trunk

        Show
        Alejandro Abdelnur added a comment - patch works against 22, needs to work against trunk
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12477709/MAPREDUCE-2457-1.patch
        against trunk revision 1097656.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these core unit tests:
        org.apache.hadoop.mapred.TestDebugScript

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/196//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/196//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/196//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12477709/MAPREDUCE-2457-1.patch against trunk revision 1097656. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.mapred.TestDebugScript -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/196//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/196//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/196//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        Patch seems good, but TestFairSchedulerSystem is already failing in trunk at the moment for another reason (not sure). Let me look into that before we commit this.

        Show
        Todd Lipcon added a comment - Patch seems good, but TestFairSchedulerSystem is already failing in trunk at the moment for another reason (not sure). Let me look into that before we commit this.
        Hide
        Aaron T. Myers added a comment -

        Since we're now adding the concept of a "primary group," I wonder if we shouldn't make this explicit, rather than just pick the first from the list. i.e. change GroupMappingServiceProvider to add a public String getPrimaryGroup(String user) method, and add a method to UserGroupInformation to get the primary group of the user.

        Thoughts?

        Show
        Aaron T. Myers added a comment - Since we're now adding the concept of a "primary group," I wonder if we shouldn't make this explicit, rather than just pick the first from the list. i.e. change GroupMappingServiceProvider to add a public String getPrimaryGroup(String user) method, and add a method to UserGroupInformation to get the primary group of the user. Thoughts?
        Hide
        Alejandro Abdelnur added a comment -

        Yes, the idea of primary group is good. We should make a new JIRA for it.

        Show
        Alejandro Abdelnur added a comment - Yes, the idea of primary group is good. We should make a new JIRA for it.
        Hide
        Todd Lipcon added a comment -

        Committed to 22 and trunk. Thanks Alejandro!

        Show
        Todd Lipcon added a comment - Committed to 22 and trunk. Thanks Alejandro!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-22-branch #44 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/44/)
        MAPREDUCE-2457. Job submission should inject group.name on the JobTracker. Contributed by Alejandro Abdelnur.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #44 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/44/ ) MAPREDUCE-2457 . Job submission should inject group.name on the JobTracker. Contributed by Alejandro Abdelnur.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #669 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/669/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #669 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/669/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #656 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/656/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #656 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/656/ )

          People

          • Assignee:
            Alejandro Abdelnur
            Reporter:
            Alejandro Abdelnur
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development