Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-563 Security features for Map/Reduce
  3. MAPREDUCE-1432

Add the hooks in JobTracker and TaskTracker to load tokens from the token cache into the user's UGI

    Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: None
    • Labels:
      None

      Description

      Related to HADOOP-6520. Here it is about putting hooks in the JobTracker/TaskTracker for loading tokens in the user's UGI. This is required when job files are copied from the HDFS on behalf of the user.

      1. MAPREDUCE-1432-BP20-2.patch
        32 kB
        Boris Shkolnik
      2. MAPREDUCE-1432-BP20.patch
        36 kB
        Boris Shkolnik
      3. 1432.2.patch
        36 kB
        Devaraj Das
      4. 1432.1.patch
        36 kB
        Devaraj Das
      5. 1432.patch
        9 kB
        Devaraj Das

        Activity

        Hide
        Boris Shkolnik added a comment -

        Patch for earlier version of Hadoop. Not for commit here.

        Show
        Boris Shkolnik added a comment - Patch for earlier version of Hadoop. Not for commit here.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #225 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/225/ )
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #222 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/222/)
        . Adds hooks in the jobtracker and tasktracker for loading the tokens in the user's ugi. This is required for the copying of files from the hdfs. Contributed by Devaraj Das.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #222 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/222/ ) . Adds hooks in the jobtracker and tasktracker for loading the tokens in the user's ugi. This is required for the copying of files from the hdfs. Contributed by Devaraj Das.
        Hide
        Devaraj Das added a comment -

        I just committed this.

        Show
        Devaraj Das added a comment - I just committed this.
        Hide
        Jakob Homan added a comment -

        +1 on the updated patch. These will fix the NPEs.

        Show
        Jakob Homan added a comment - +1 on the updated patch. These will fix the NPEs.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12431879/1432.1.patch
        against trunk revision 904717.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 12 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/420/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/420/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/420/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/420/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12431879/1432.1.patch against trunk revision 904717. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/420/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/420/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/420/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/420/console This message is automatically generated.
        Hide
        Devaraj Das added a comment -

        All tests and test-passed with this patch

        I meant tests and test-patch.

        Show
        Devaraj Das added a comment - All tests and test-passed with this patch I meant tests and test-patch.
        Hide
        Devaraj Das added a comment -

        All tests and test-passed with this patch. The fixes are checks for null for the TokenStorage/Token objects in the JobInProgress and TaskTracker. Some unit tests pass them as null and without the checks those tests fail with NPE.

        The relevant portion of the diff w.r.t the earlier patch is:

        > @@ -377,16 +378,22 @@
        > this.startTime = System.currentTimeMillis();
        >
        > this.localFs = jobtracker.getLocalFileSystem();
        > -
        > + this.tokenStorage = ts;
        97,98c102,105
        < + for (Token<? extends TokenIdentifier> token : ts.getAllTokens()) {
        < + ugi.addToken(token);

        > + if (ts != null) {
        > + for (Token<? extends TokenIdentifier> token : ts.getAllTokens())

        { > + ugi.addToken(token); > + }

        108a116,124
        > @@ -449,8 +456,6 @@
        > JobContext.SPECULATIVECAP,0.1f);
        > this.slowNodeThreshold = conf.getFloat(
        > JobContext.SPECULATIVE_SLOWNODE_THRESHOLD,1.0f);
        > - this.tokenStorage = ts;
        > -
        > }
        >
        > /**
        247c263
        < @@ -993,7 +996,17 @@

        > @@ -993,7 +996,19 @@
        257,258c273,276
        < + (Token<JobTokenIdentifier>)TokenCache.getJobToken(ts);
        < + getJobTokenSecretManager().addTokenForJob(jobId.toString(), jt);

        > + (Token<JobTokenIdentifier>)TokenCache.getJobToken(ts);
        > + if (jt != null)

        { //could be null in the case of some unit tests > + getJobTokenSecretManager().addTokenForJob(jobId.toString(), jt); > + }
        Show
        Devaraj Das added a comment - All tests and test-passed with this patch. The fixes are checks for null for the TokenStorage/Token objects in the JobInProgress and TaskTracker. Some unit tests pass them as null and without the checks those tests fail with NPE. The relevant portion of the diff w.r.t the earlier patch is: > @@ -377,16 +378,22 @@ > this.startTime = System.currentTimeMillis(); > > this.localFs = jobtracker.getLocalFileSystem(); > - > + this.tokenStorage = ts; 97,98c102,105 < + for (Token<? extends TokenIdentifier> token : ts.getAllTokens()) { < + ugi.addToken(token); — > + if (ts != null) { > + for (Token<? extends TokenIdentifier> token : ts.getAllTokens()) { > + ugi.addToken(token); > + } 108a116,124 > @@ -449,8 +456,6 @@ > JobContext.SPECULATIVECAP,0.1f); > this.slowNodeThreshold = conf.getFloat( > JobContext.SPECULATIVE_SLOWNODE_THRESHOLD,1.0f); > - this.tokenStorage = ts; > - > } > > /** 247c263 < @@ -993,7 +996,17 @@ — > @@ -993,7 +996,19 @@ 257,258c273,276 < + (Token<JobTokenIdentifier>)TokenCache.getJobToken(ts); < + getJobTokenSecretManager().addTokenForJob(jobId.toString(), jt); — > + (Token<JobTokenIdentifier>)TokenCache.getJobToken(ts); > + if (jt != null) { //could be null in the case of some unit tests > + getJobTokenSecretManager().addTokenForJob(jobId.toString(), jt); > + }
        Hide
        Owen O'Malley added a comment -

        This looks good.

        +1

        Show
        Owen O'Malley added a comment - This looks good. +1
        Hide
        Devaraj Das added a comment -

        Trying hudson

        Show
        Devaraj Das added a comment - Trying hudson
        Hide
        Devaraj Das added a comment -

        Attaching a patch for review. The patch adds hooks in the JobInProgress/TaskTracker for the UGI. It also removes the TokenStorage/TestTokenStorage classes. The TokenCache class is cleaned up a little..

        Show
        Devaraj Das added a comment - Attaching a patch for review. The patch adds hooks in the JobInProgress/TaskTracker for the UGI. It also removes the TokenStorage/TestTokenStorage classes. The TokenCache class is cleaned up a little..
        Hide
        Devaraj Das added a comment -

        Attaching a patch that puts the hooks. I will refactor the tokencache out of mapreduce to common in a follow up patch.

        Show
        Devaraj Das added a comment - Attaching a patch that puts the hooks. I will refactor the tokencache out of mapreduce to common in a follow up patch.

          People

          • Assignee:
            Devaraj Das
            Reporter:
            Devaraj Das
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development