Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-6520

UGI should load tokens from the environment

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.22.0
    • 0.21.0
    • None
    • None
    • Reviewed

    Description

      For MapReduce tasks, we need not only the task itself, but child processes to get the delegation tokens, therefore we need an environment variable that if set loads a token cache into the login UGI. The task tracker can set this property so that tasks (and child processes of the tasks, such as streaming and pipes) have the job's tokens automatically.

      Attachments

        1. h-6520.1.patch
          2 kB
          Devaraj Das
        2. h-6520.2.patch
          2 kB
          Devaraj Das
        3. h-6520.3.patch
          12 kB
          Devaraj Das
        4. h-6520.4.patch
          12 kB
          Devaraj Das
        5. h-6520.patch
          2 kB
          Devaraj Das
        6. HADOOP-6520-0_20.2.patch
          12 kB
          Jitendra Nath Pandey

        Issue Links

          Activity

            ddas Devaraj Das added a comment -

            This environment variable exists - JOB_TOKEN_FILE. In this jira, we should read the contents of the file via TokenCache's APIs and load the tokens in the UGI (similar to the way job-token is loaded in Child.java). In the JobInProgress, the tokens already exist in memory and it should be easy to add it to the ugi used for copying the jobconf file from the hdfs. In the tasktracker, ditto (for localizing the job's files).
            For tasks (and for other processes like an MR Streaming app running a hdfs command from within the tasks), if we have a static block in the UGI to look for the env variable, and load the token cache using the TokenCache's APIs, it should work. Also, when the env var is defined, we should use simple authentication for login.
            Thoughts?

            ddas Devaraj Das added a comment - This environment variable exists - JOB_TOKEN_FILE. In this jira, we should read the contents of the file via TokenCache's APIs and load the tokens in the UGI (similar to the way job-token is loaded in Child.java). In the JobInProgress, the tokens already exist in memory and it should be easy to add it to the ugi used for copying the jobconf file from the hdfs. In the tasktracker, ditto (for localizing the job's files). For tasks (and for other processes like an MR Streaming app running a hdfs command from within the tasks), if we have a static block in the UGI to look for the env variable, and load the token cache using the TokenCache's APIs, it should work. Also, when the env var is defined, we should use simple authentication for login. Thoughts?
            ddas Devaraj Das added a comment -

            Attaching a patch for common. For the mapreduce part, we need to have a separate jira. This patch disables kerberos login when the env var is set. In the method getLoginUser in UserGroupInformation, the token file is read and tokens are loaded.
            Thoughts?

            ddas Devaraj Das added a comment - Attaching a patch for common. For the mapreduce part, we need to have a separate jira. This patch disables kerberos login when the env var is set. In the method getLoginUser in UserGroupInformation, the token file is read and tokens are loaded. Thoughts?
            omalley Owen O'Malley added a comment -

            I'd flip the test around so that we don't load tokens if we are in the simple authentication.

            It also should leave the kerberos flag turned on even if there is a token file.

            I'd also suggest changing the name to HADOOP_TOKEN_FILE and making it a static final string constant.

            omalley Owen O'Malley added a comment - I'd flip the test around so that we don't load tokens if we are in the simple authentication. It also should leave the kerberos flag turned on even if there is a token file. I'd also suggest changing the name to HADOOP_TOKEN_FILE and making it a static final string constant.
            ddas Devaraj Das added a comment -

            Addressing Owen's comments.

            ddas Devaraj Das added a comment - Addressing Owen's comments.
            omalley Owen O'Malley added a comment -

            This looks good, but:
            1. unit test
            2. a line is too long (> 80 chrs)

            omalley Owen O'Malley added a comment - This looks good, but: 1. unit test 2. a line is too long (> 80 chrs)
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12431836/h-6520.1.patch
            against trunk revision 904339.

            +1 @author. The patch does not contain any @author tags.

            -1 tests included. The patch doesn't appear to include any new or modified tests.
            Please justify why no new tests are needed for this patch.
            Also please list what manual steps were performed to verify this patch.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/testReport/
            Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/artifact/trunk/build/test/checkstyle-errors.html
            Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431836/h-6520.1.patch against trunk revision 904339. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/console This message is automatically generated.
            ddas Devaraj Das added a comment -

            Fixed the line length issue. I tested the patch manually. Difficult to write a test for this since this patch relies on env variables being exported.

            ddas Devaraj Das added a comment - Fixed the line length issue. I tested the patch manually. Difficult to write a test for this since this patch relies on env variables being exported.
            omalley Owen O'Malley added a comment -

            I agree on the unit tests.

            Shouldn't this be using the code in TokenStorage from MAPREDUCE-1338? Replicating the code to read the file into this class seems wrong. Unfortunately, we would need to move the TokenStorage code down into Common...

            omalley Owen O'Malley added a comment - I agree on the unit tests. Shouldn't this be using the code in TokenStorage from MAPREDUCE-1338 ? Replicating the code to read the file into this class seems wrong. Unfortunately, we would need to move the TokenStorage code down into Common...
            ddas Devaraj Das added a comment -

            Shouldn't this be using the code in TokenStorage from MAPREDUCE-1338? Replicating the code to read the file into this class seems wrong. Unfortunately, we would need to move the TokenStorage code down into Common...

            That's true. I was thinking about doing it in a future jira since it involves couple of mapreduce jiras - MAPREDUCE-1383 & MAPREDUCE-1338, but okay I will put that effort in this jira & MAPREDUCE-1432.

            ddas Devaraj Das added a comment - Shouldn't this be using the code in TokenStorage from MAPREDUCE-1338 ? Replicating the code to read the file into this class seems wrong. Unfortunately, we would need to move the TokenStorage code down into Common... That's true. I was thinking about doing it in a future jira since it involves couple of mapreduce jiras - MAPREDUCE-1383 & MAPREDUCE-1338 , but okay I will put that effort in this jira & MAPREDUCE-1432 .
            ddas Devaraj Das added a comment -

            Ok, this patch adds TokenStorage class in Common. Not much has changed since my earlier patch other than creation of the new files TokenStorage.java & TestTokenStorage.java, and invocation of TokenStorage.readToken directly from the UserGroupInformation.java..

            ddas Devaraj Das added a comment - Ok, this patch adds TokenStorage class in Common. Not much has changed since my earlier patch other than creation of the new files TokenStorage.java & TestTokenStorage.java, and invocation of TokenStorage.readToken directly from the UserGroupInformation.java..
            hadoopqa Hadoop QA added a comment -

            -1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12431861/h-6520.3.patch
            against trunk revision 904339.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 3 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            -1 javac. The applied patch generated 1030 javac compiler warnings (more than the trunk's current 1028 warnings).

            +1 findbugs. The patch does not introduce any new Findbugs warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/testReport/
            Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/artifact/trunk/build/test/checkstyle-errors.html
            Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431861/h-6520.3.patch against trunk revision 904339. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 1030 javac compiler warnings (more than the trunk's current 1028 warnings). +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/console This message is automatically generated.
            omalley Owen O'Malley added a comment -

            +1

            I don't see the JavaDoc warning when I compile, so I don't know what Hudson saw...

            omalley Owen O'Malley added a comment - +1 I don't see the JavaDoc warning when I compile, so I don't know what Hudson saw...
            ddas Devaraj Das added a comment -

            Fixing the javac warnings (i think it was coming out of TestTokenStorage.java due to the use of a deprecated class). Also, changed the Interface audience for TokenStorage to be only MapReduce.

            ddas Devaraj Das added a comment - Fixing the javac warnings (i think it was coming out of TestTokenStorage.java due to the use of a deprecated class). Also, changed the Interface audience for TokenStorage to be only MapReduce.
            hadoopqa Hadoop QA added a comment -

            +1 overall. Here are the results of testing the latest attachment
            http://issues.apache.org/jira/secure/attachment/12431871/h-6520.4.patch
            against trunk revision 904339.

            +1 @author. The patch does not contain any @author tags.

            +1 tests included. The patch appears to include 3 new or modified tests.

            +1 javadoc. The javadoc tool did not generate any warning messages.

            +1 javac. The applied patch does not increase the total number of javac compiler warnings.

            +1 findbugs. The patch does not introduce any new Findbugs warnings.

            +1 release audit. The applied patch does not increase the total number of release audit warnings.

            +1 core tests. The patch passed core unit tests.

            +1 contrib tests. The patch passed contrib unit tests.

            Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/testReport/
            Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
            Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/artifact/trunk/build/test/checkstyle-errors.html
            Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/console

            This message is automatically generated.

            hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431871/h-6520.4.patch against trunk revision 904339. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/console This message is automatically generated.
            omalley Owen O'Malley added a comment -

            +1

            omalley Owen O'Malley added a comment - +1
            ddas Devaraj Das added a comment -

            I just committed this.

            ddas Devaraj Das added a comment - I just committed this.
            hudson Hudson added a comment -

            Integrated in Hadoop-Common-trunk-Commit #152 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/152/)
            . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.
            . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.

            hudson Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #152 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/152/ ) . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das. . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.
            hudson Hudson added a comment -

            Integrated in Hadoop-Common-trunk #238 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/238/)
            . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.
            . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.

            hudson Hudson added a comment - Integrated in Hadoop-Common-trunk #238 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/238/ ) . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das. . Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.

            Patch for Hadoop-20 added.

            jnp Jitendra Nath Pandey added a comment - Patch for Hadoop-20 added.

            People

              ddas Devaraj Das
              omalley Owen O'Malley
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: