Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-6520

UGI should load tokens from the environment

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      For MapReduce tasks, we need not only the task itself, but child processes to get the delegation tokens, therefore we need an environment variable that if set loads a token cache into the login UGI. The task tracker can set this property so that tasks (and child processes of the tasks, such as streaming and pipes) have the job's tokens automatically.

      1. h-6520.1.patch
        2 kB
        Devaraj Das
      2. h-6520.2.patch
        2 kB
        Devaraj Das
      3. h-6520.3.patch
        12 kB
        Devaraj Das
      4. h-6520.4.patch
        12 kB
        Devaraj Das
      5. h-6520.patch
        2 kB
        Devaraj Das
      6. HADOOP-6520-0_20.2.patch
        12 kB
        Jitendra Nath Pandey

        Issue Links

          Activity

          Hide
          devaraj Devaraj Das added a comment -

          This environment variable exists - JOB_TOKEN_FILE. In this jira, we should read the contents of the file via TokenCache's APIs and load the tokens in the UGI (similar to the way job-token is loaded in Child.java). In the JobInProgress, the tokens already exist in memory and it should be easy to add it to the ugi used for copying the jobconf file from the hdfs. In the tasktracker, ditto (for localizing the job's files).
          For tasks (and for other processes like an MR Streaming app running a hdfs command from within the tasks), if we have a static block in the UGI to look for the env variable, and load the token cache using the TokenCache's APIs, it should work. Also, when the env var is defined, we should use simple authentication for login.
          Thoughts?

          Show
          devaraj Devaraj Das added a comment - This environment variable exists - JOB_TOKEN_FILE. In this jira, we should read the contents of the file via TokenCache's APIs and load the tokens in the UGI (similar to the way job-token is loaded in Child.java). In the JobInProgress, the tokens already exist in memory and it should be easy to add it to the ugi used for copying the jobconf file from the hdfs. In the tasktracker, ditto (for localizing the job's files). For tasks (and for other processes like an MR Streaming app running a hdfs command from within the tasks), if we have a static block in the UGI to look for the env variable, and load the token cache using the TokenCache's APIs, it should work. Also, when the env var is defined, we should use simple authentication for login. Thoughts?
          Hide
          devaraj Devaraj Das added a comment -

          Attaching a patch for common. For the mapreduce part, we need to have a separate jira. This patch disables kerberos login when the env var is set. In the method getLoginUser in UserGroupInformation, the token file is read and tokens are loaded.
          Thoughts?

          Show
          devaraj Devaraj Das added a comment - Attaching a patch for common. For the mapreduce part, we need to have a separate jira. This patch disables kerberos login when the env var is set. In the method getLoginUser in UserGroupInformation, the token file is read and tokens are loaded. Thoughts?
          Hide
          owen.omalley Owen O'Malley added a comment -

          I'd flip the test around so that we don't load tokens if we are in the simple authentication.

          It also should leave the kerberos flag turned on even if there is a token file.

          I'd also suggest changing the name to HADOOP_TOKEN_FILE and making it a static final string constant.

          Show
          owen.omalley Owen O'Malley added a comment - I'd flip the test around so that we don't load tokens if we are in the simple authentication. It also should leave the kerberos flag turned on even if there is a token file. I'd also suggest changing the name to HADOOP_TOKEN_FILE and making it a static final string constant.
          Hide
          devaraj Devaraj Das added a comment -

          Addressing Owen's comments.

          Show
          devaraj Devaraj Das added a comment - Addressing Owen's comments.
          Hide
          owen.omalley Owen O'Malley added a comment -

          This looks good, but:
          1. unit test
          2. a line is too long (> 80 chrs)

          Show
          owen.omalley Owen O'Malley added a comment - This looks good, but: 1. unit test 2. a line is too long (> 80 chrs)
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12431836/h-6520.1.patch
          against trunk revision 904339.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431836/h-6520.1.patch against trunk revision 904339. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/console This message is automatically generated.
          Hide
          devaraj Devaraj Das added a comment -

          Fixed the line length issue. I tested the patch manually. Difficult to write a test for this since this patch relies on env variables being exported.

          Show
          devaraj Devaraj Das added a comment - Fixed the line length issue. I tested the patch manually. Difficult to write a test for this since this patch relies on env variables being exported.
          Hide
          owen.omalley Owen O'Malley added a comment -

          I agree on the unit tests.

          Shouldn't this be using the code in TokenStorage from MAPREDUCE-1338? Replicating the code to read the file into this class seems wrong. Unfortunately, we would need to move the TokenStorage code down into Common...

          Show
          owen.omalley Owen O'Malley added a comment - I agree on the unit tests. Shouldn't this be using the code in TokenStorage from MAPREDUCE-1338 ? Replicating the code to read the file into this class seems wrong. Unfortunately, we would need to move the TokenStorage code down into Common...
          Hide
          devaraj Devaraj Das added a comment -

          Shouldn't this be using the code in TokenStorage from MAPREDUCE-1338? Replicating the code to read the file into this class seems wrong. Unfortunately, we would need to move the TokenStorage code down into Common...

          That's true. I was thinking about doing it in a future jira since it involves couple of mapreduce jiras - MAPREDUCE-1383 & MAPREDUCE-1338, but okay I will put that effort in this jira & MAPREDUCE-1432.

          Show
          devaraj Devaraj Das added a comment - Shouldn't this be using the code in TokenStorage from MAPREDUCE-1338 ? Replicating the code to read the file into this class seems wrong. Unfortunately, we would need to move the TokenStorage code down into Common... That's true. I was thinking about doing it in a future jira since it involves couple of mapreduce jiras - MAPREDUCE-1383 & MAPREDUCE-1338 , but okay I will put that effort in this jira & MAPREDUCE-1432 .
          Hide
          devaraj Devaraj Das added a comment -

          Ok, this patch adds TokenStorage class in Common. Not much has changed since my earlier patch other than creation of the new files TokenStorage.java & TestTokenStorage.java, and invocation of TokenStorage.readToken directly from the UserGroupInformation.java..

          Show
          devaraj Devaraj Das added a comment - Ok, this patch adds TokenStorage class in Common. Not much has changed since my earlier patch other than creation of the new files TokenStorage.java & TestTokenStorage.java, and invocation of TokenStorage.readToken directly from the UserGroupInformation.java..
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12431861/h-6520.3.patch
          against trunk revision 904339.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 javac. The applied patch generated 1030 javac compiler warnings (more than the trunk's current 1028 warnings).

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431861/h-6520.3.patch against trunk revision 904339. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 1030 javac compiler warnings (more than the trunk's current 1028 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/console This message is automatically generated.
          Hide
          owen.omalley Owen O'Malley added a comment -

          +1

          I don't see the JavaDoc warning when I compile, so I don't know what Hudson saw...

          Show
          owen.omalley Owen O'Malley added a comment - +1 I don't see the JavaDoc warning when I compile, so I don't know what Hudson saw...
          Hide
          devaraj Devaraj Das added a comment -

          Fixing the javac warnings (i think it was coming out of TestTokenStorage.java due to the use of a deprecated class). Also, changed the Interface audience for TokenStorage to be only MapReduce.

          Show
          devaraj Devaraj Das added a comment - Fixing the javac warnings (i think it was coming out of TestTokenStorage.java due to the use of a deprecated class). Also, changed the Interface audience for TokenStorage to be only MapReduce.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12431871/h-6520.4.patch
          against trunk revision 904339.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431871/h-6520.4.patch against trunk revision 904339. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/console This message is automatically generated.
          Hide
          owen.omalley Owen O'Malley added a comment -

          +1

          Show
          owen.omalley Owen O'Malley added a comment - +1
          Hide
          devaraj Devaraj Das added a comment -

          I just committed this.

          Show
          devaraj Devaraj Das added a comment - I just committed this.
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #152 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/152/)
          . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.
          . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.

          Show
          hudson Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #152 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/152/ ) . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das. . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.
          Hide
          hudson Hudson added a comment -

          Integrated in Hadoop-Common-trunk #238 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/238/)
          . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.
          . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.

          Show
          hudson Hudson added a comment - Integrated in Hadoop-Common-trunk #238 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/238/ ) . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das. . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.
          Hide
          jnp Jitendra Nath Pandey added a comment -

          Patch for Hadoop-20 added.

          Show
          jnp Jitendra Nath Pandey added a comment - Patch for Hadoop-20 added.

            People

            • Assignee:
              devaraj Devaraj Das
              Reporter:
              owen.omalley Owen O'Malley
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development