Hadoop Common
  1. Hadoop Common
  2. HADOOP-6520

UGI should load tokens from the environment

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      For MapReduce tasks, we need not only the task itself, but child processes to get the delegation tokens, therefore we need an environment variable that if set loads a token cache into the login UGI. The task tracker can set this property so that tasks (and child processes of the tasks, such as streaming and pipes) have the job's tokens automatically.

      1. h-6520.patch
        2 kB
        Devaraj Das
      2. h-6520.1.patch
        2 kB
        Devaraj Das
      3. h-6520.2.patch
        2 kB
        Devaraj Das
      4. h-6520.3.patch
        12 kB
        Devaraj Das
      5. h-6520.4.patch
        12 kB
        Devaraj Das
      6. HADOOP-6520-0_20.2.patch
        12 kB
        Jitendra Nath Pandey

        Activity

        Hide
        Devaraj Das added a comment -

        This environment variable exists - JOB_TOKEN_FILE. In this jira, we should read the contents of the file via TokenCache's APIs and load the tokens in the UGI (similar to the way job-token is loaded in Child.java). In the JobInProgress, the tokens already exist in memory and it should be easy to add it to the ugi used for copying the jobconf file from the hdfs. In the tasktracker, ditto (for localizing the job's files).
        For tasks (and for other processes like an MR Streaming app running a hdfs command from within the tasks), if we have a static block in the UGI to look for the env variable, and load the token cache using the TokenCache's APIs, it should work. Also, when the env var is defined, we should use simple authentication for login.
        Thoughts?

        Show
        Devaraj Das added a comment - This environment variable exists - JOB_TOKEN_FILE. In this jira, we should read the contents of the file via TokenCache's APIs and load the tokens in the UGI (similar to the way job-token is loaded in Child.java). In the JobInProgress, the tokens already exist in memory and it should be easy to add it to the ugi used for copying the jobconf file from the hdfs. In the tasktracker, ditto (for localizing the job's files). For tasks (and for other processes like an MR Streaming app running a hdfs command from within the tasks), if we have a static block in the UGI to look for the env variable, and load the token cache using the TokenCache's APIs, it should work. Also, when the env var is defined, we should use simple authentication for login. Thoughts?
        Hide
        Devaraj Das added a comment -

        Attaching a patch for common. For the mapreduce part, we need to have a separate jira. This patch disables kerberos login when the env var is set. In the method getLoginUser in UserGroupInformation, the token file is read and tokens are loaded.
        Thoughts?

        Show
        Devaraj Das added a comment - Attaching a patch for common. For the mapreduce part, we need to have a separate jira. This patch disables kerberos login when the env var is set. In the method getLoginUser in UserGroupInformation, the token file is read and tokens are loaded. Thoughts?
        Hide
        Owen O'Malley added a comment -

        I'd flip the test around so that we don't load tokens if we are in the simple authentication.

        It also should leave the kerberos flag turned on even if there is a token file.

        I'd also suggest changing the name to HADOOP_TOKEN_FILE and making it a static final string constant.

        Show
        Owen O'Malley added a comment - I'd flip the test around so that we don't load tokens if we are in the simple authentication. It also should leave the kerberos flag turned on even if there is a token file. I'd also suggest changing the name to HADOOP_TOKEN_FILE and making it a static final string constant.
        Hide
        Devaraj Das added a comment -

        Addressing Owen's comments.

        Show
        Devaraj Das added a comment - Addressing Owen's comments.
        Hide
        Owen O'Malley added a comment -

        This looks good, but:
        1. unit test
        2. a line is too long (> 80 chrs)

        Show
        Owen O'Malley added a comment - This looks good, but: 1. unit test 2. a line is too long (> 80 chrs)
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12431836/h-6520.1.patch
        against trunk revision 904339.

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431836/h-6520.1.patch against trunk revision 904339. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/console This message is automatically generated.
        Hide
        Devaraj Das added a comment -

        Fixed the line length issue. I tested the patch manually. Difficult to write a test for this since this patch relies on env variables being exported.

        Show
        Devaraj Das added a comment - Fixed the line length issue. I tested the patch manually. Difficult to write a test for this since this patch relies on env variables being exported.
        Hide
        Owen O'Malley added a comment -

        I agree on the unit tests.

        Shouldn't this be using the code in TokenStorage from MAPREDUCE-1338? Replicating the code to read the file into this class seems wrong. Unfortunately, we would need to move the TokenStorage code down into Common...

        Show
        Owen O'Malley added a comment - I agree on the unit tests. Shouldn't this be using the code in TokenStorage from MAPREDUCE-1338 ? Replicating the code to read the file into this class seems wrong. Unfortunately, we would need to move the TokenStorage code down into Common...
        Hide
        Devaraj Das added a comment -

        Shouldn't this be using the code in TokenStorage from MAPREDUCE-1338? Replicating the code to read the file into this class seems wrong. Unfortunately, we would need to move the TokenStorage code down into Common...

        That's true. I was thinking about doing it in a future jira since it involves couple of mapreduce jiras - MAPREDUCE-1383 & MAPREDUCE-1338, but okay I will put that effort in this jira & MAPREDUCE-1432.

        Show
        Devaraj Das added a comment - Shouldn't this be using the code in TokenStorage from MAPREDUCE-1338 ? Replicating the code to read the file into this class seems wrong. Unfortunately, we would need to move the TokenStorage code down into Common... That's true. I was thinking about doing it in a future jira since it involves couple of mapreduce jiras - MAPREDUCE-1383 & MAPREDUCE-1338 , but okay I will put that effort in this jira & MAPREDUCE-1432 .
        Hide
        Devaraj Das added a comment -

        Ok, this patch adds TokenStorage class in Common. Not much has changed since my earlier patch other than creation of the new files TokenStorage.java & TestTokenStorage.java, and invocation of TokenStorage.readToken directly from the UserGroupInformation.java..

        Show
        Devaraj Das added a comment - Ok, this patch adds TokenStorage class in Common. Not much has changed since my earlier patch other than creation of the new files TokenStorage.java & TestTokenStorage.java, and invocation of TokenStorage.readToken directly from the UserGroupInformation.java..
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12431861/h-6520.3.patch
        against trunk revision 904339.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        -1 javac. The applied patch generated 1030 javac compiler warnings (more than the trunk's current 1028 warnings).

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431861/h-6520.3.patch against trunk revision 904339. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 1030 javac compiler warnings (more than the trunk's current 1028 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/console This message is automatically generated.
        Hide
        Owen O'Malley added a comment -

        +1

        I don't see the JavaDoc warning when I compile, so I don't know what Hudson saw...

        Show
        Owen O'Malley added a comment - +1 I don't see the JavaDoc warning when I compile, so I don't know what Hudson saw...
        Hide
        Devaraj Das added a comment -

        Fixing the javac warnings (i think it was coming out of TestTokenStorage.java due to the use of a deprecated class). Also, changed the Interface audience for TokenStorage to be only MapReduce.

        Show
        Devaraj Das added a comment - Fixing the javac warnings (i think it was coming out of TestTokenStorage.java due to the use of a deprecated class). Also, changed the Interface audience for TokenStorage to be only MapReduce.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12431871/h-6520.4.patch
        against trunk revision 904339.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431871/h-6520.4.patch against trunk revision 904339. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/console This message is automatically generated.
        Hide
        Owen O'Malley added a comment -

        +1

        Show
        Owen O'Malley added a comment - +1
        Hide
        Devaraj Das added a comment -

        I just committed this.

        Show
        Devaraj Das added a comment - I just committed this.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #152 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/152/)
        . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.
        . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #152 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/152/ ) . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das. . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk #238 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/238/)
        . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.
        . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.

        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk #238 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/238/ ) . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das. . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.
        Hide
        Jitendra Nath Pandey added a comment -

        Patch for Hadoop-20 added.

        Show
        Jitendra Nath Pandey added a comment - Patch for Hadoop-20 added.

          People

          • Assignee:
            Devaraj Das
            Reporter:
            Owen O'Malley
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development