Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1532

Delegation token is obtained as the superuser

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: job submission, security
    • Labels:
      None

      Description

      When the UserGroupInformation.doAs is invoked for proxy users, the delegation token is incorrectly obtained as the real user.

      1. 1532-bp20.1.patch
        19 kB
        Devaraj Das
      2. 1532-bp20.2.patch
        20 kB
        Devaraj Das
      3. 1532-bp20.4.patch
        20 kB
        Devaraj Das
      4. 1532-bp20.4.1.patch
        2 kB
        Devaraj Das
      5. 1532-bp20.4.2.patch
        1 kB
        Devaraj Das
      6. ASF.LICENSE.NOT.GRANTED--1532.1.patch
        12 kB
        Devaraj Das
      7. ASF.LICENSE.NOT.GRANTED--1532.2.patch
        13 kB
        Devaraj Das

        Activity

        Hide
        Devaraj Das added a comment -

        This is a patch for Y20. The changes:
        1) The JobClient and the Job constructors cache the UserGroupInformation. That UGI is used for getting the FileSystem in the JobClient and the Job classes.
        2) The DelegationTokenRenewal is changed a bit. It now maintains a list of delegation tokens as opposed to a map from JobID to the tokens. This is required to support the Oozie use case better. Otherwise, we will have duplicate tokens for those cases where a task of one job launches another job. That second job uses the same tokens.

        Show
        Devaraj Das added a comment - This is a patch for Y20. The changes: 1) The JobClient and the Job constructors cache the UserGroupInformation. That UGI is used for getting the FileSystem in the JobClient and the Job classes. 2) The DelegationTokenRenewal is changed a bit. It now maintains a list of delegation tokens as opposed to a map from JobID to the tokens. This is required to support the Oozie use case better. Otherwise, we will have duplicate tokens for those cases where a task of one job launches another job. That second job uses the same tokens.
        Hide
        Devaraj Das added a comment -

        Addresses some bugs in the earlier patch.

        Show
        Devaraj Das added a comment - Addresses some bugs in the earlier patch.
        Hide
        Boris Shkolnik added a comment -

        I've reviewed DelegationTokenRenewal related parts.
        Changes to DelegationTokenRenewal.java look fine.
        In TestDTR:
        1. I don't think it is a good idea to sleep for 10 seconds every time. If you feel it can take long then the currently specified time - increase the number of attempts, but not the attempt span. So in the good case we don't slow tests too much.
        2. the attempt thing is really needed only in the first part of the test. In the second part we test negative case - it should never be renewed.
        So one try or 3 seconds should suffice.

        Show
        Boris Shkolnik added a comment - I've reviewed DelegationTokenRenewal related parts. Changes to DelegationTokenRenewal.java look fine. In TestDTR: 1. I don't think it is a good idea to sleep for 10 seconds every time. If you feel it can take long then the currently specified time - increase the number of attempts, but not the attempt span. So in the good case we don't slow tests too much. 2. the attempt thing is really needed only in the first part of the test. In the second part we test negative case - it should never be renewed. So one try or 3 seconds should suffice.
        Hide
        Devaraj Das added a comment -

        Attaching a patch that addresses the concerns. (trunk patch coming soon)

        Show
        Devaraj Das added a comment - Attaching a patch that addresses the concerns. (trunk patch coming soon)
        Hide
        Boris Shkolnik added a comment -

        I've reviewed DelegationTokenRenewal related parts.
        Changes to DelegationTokenRenewal.java look fine.
        In TestDTR:
        1. I don't think it is a good idea to sleep for 10 seconds every time. If you feel it can take long then the currently specified time - increase the number of attempts, but not the attempt span. So in the good case we don't slow tests too much.
        2. the attempt thing is really needed only in the first part of the test. In the second part we test negative case - it should never be renewed.
        So one try or 3 seconds should suffice.

        Show
        Boris Shkolnik added a comment - I've reviewed DelegationTokenRenewal related parts. Changes to DelegationTokenRenewal.java look fine. In TestDTR: 1. I don't think it is a good idea to sleep for 10 seconds every time. If you feel it can take long then the currently specified time - increase the number of attempts, but not the attempt span. So in the good case we don't slow tests too much. 2. the attempt thing is really needed only in the first part of the test. In the second part we test negative case - it should never be renewed. So one try or 3 seconds should suffice.
        Hide
        Devaraj Das added a comment -

        Patch has some javadoc and a clearer exception message in jobInProgress's constructor when it detects that the authenticated user is different from the user in the configuration.

        Show
        Devaraj Das added a comment - Patch has some javadoc and a clearer exception message in jobInProgress's constructor when it detects that the authenticated user is different from the user in the configuration.
        Hide
        Devaraj Das added a comment -

        A bit updated version of the same patch.

        Show
        Devaraj Das added a comment - A bit updated version of the same patch.
        Hide
        Devaraj Das added a comment -

        The attached patch is for trunk

        Show
        Devaraj Das added a comment - The attached patch is for trunk
        Hide
        Devaraj Das added a comment -

        A slightly updated patch. An equivalent patch for y20s has been manually tested.

        Show
        Devaraj Das added a comment - A slightly updated patch. An equivalent patch for y20s has been manually tested.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12442031/1532.2.patch
        against trunk revision 935090.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/117/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/117/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/117/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/117/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12442031/1532.2.patch against trunk revision 935090. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/117/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/117/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/117/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/117/console This message is automatically generated.
        Hide
        Chris Douglas added a comment -

        +1

        Show
        Chris Douglas added a comment - +1
        Hide
        Devaraj Das added a comment -

        Thanks Chris for the review.

        On 5/5/10 7:18 PM, "Chris Douglas (JIRA)" <jira@apache.org> wrote:

        [ https://issues.apache.org/jira/browse/MAPREDUCE-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12864621#action_12864621 ]

        Chris Douglas commented on MAPREDUCE-1532:
        ------------------------------------------

        +1


        This message is automatically generated by JIRA.
        -
        You can reply to this email to add a comment to the issue online.

        Show
        Devaraj Das added a comment - Thanks Chris for the review. On 5/5/10 7:18 PM, "Chris Douglas (JIRA)" <jira@apache.org> wrote: [ https://issues.apache.org/jira/browse/MAPREDUCE-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12864621#action_12864621 ] Chris Douglas commented on MAPREDUCE-1532 : ------------------------------------------ +1 – This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
        Hide
        Devaraj Das added a comment -

        I just committed this.

        Show
        Devaraj Das added a comment - I just committed this.

          People

          • Assignee:
            Devaraj Das
            Reporter:
            Devaraj Das
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development