Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
0.22.0
-
None
-
None
-
Reviewed
Description
For MapReduce tasks, we need not only the task itself, but child processes to get the delegation tokens, therefore we need an environment variable that if set loads a token cache into the login UGI. The task tracker can set this property so that tasks (and child processes of the tasks, such as streaming and pipes) have the job's tokens automatically.
Attachments
Attachments
- h-6520.1.patch
- 2 kB
- Devaraj Das
- h-6520.2.patch
- 2 kB
- Devaraj Das
- h-6520.3.patch
- 12 kB
- Devaraj Das
- h-6520.4.patch
- 12 kB
- Devaraj Das
- h-6520.patch
- 2 kB
- Devaraj Das
- HADOOP-6520-0_20.2.patch
- 12 kB
- Jitendra Nath Pandey
Issue Links
- relates to
-
HADOOP-14808 Hadoop keychain
- Patch Available
Activity
Attaching a patch for common. For the mapreduce part, we need to have a separate jira. This patch disables kerberos login when the env var is set. In the method getLoginUser in UserGroupInformation, the token file is read and tokens are loaded.
Thoughts?
I'd flip the test around so that we don't load tokens if we are in the simple authentication.
It also should leave the kerberos flag turned on even if there is a token file.
I'd also suggest changing the name to HADOOP_TOKEN_FILE and making it a static final string constant.
This looks good, but:
1. unit test
2. a line is too long (> 80 chrs)
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12431836/h-6520.1.patch
against trunk revision 904339.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/301/console
This message is automatically generated.
Fixed the line length issue. I tested the patch manually. Difficult to write a test for this since this patch relies on env variables being exported.
I agree on the unit tests.
Shouldn't this be using the code in TokenStorage from MAPREDUCE-1338? Replicating the code to read the file into this class seems wrong. Unfortunately, we would need to move the TokenStorage code down into Common...
Shouldn't this be using the code in TokenStorage from
MAPREDUCE-1338? Replicating the code to read the file into this class seems wrong. Unfortunately, we would need to move the TokenStorage code down into Common...
That's true. I was thinking about doing it in a future jira since it involves couple of mapreduce jiras - MAPREDUCE-1383 & MAPREDUCE-1338, but okay I will put that effort in this jira & MAPREDUCE-1432.
Ok, this patch adds TokenStorage class in Common. Not much has changed since my earlier patch other than creation of the new files TokenStorage.java & TestTokenStorage.java, and invocation of TokenStorage.readToken directly from the UserGroupInformation.java..
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12431861/h-6520.3.patch
against trunk revision 904339.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The applied patch generated 1030 javac compiler warnings (more than the trunk's current 1028 warnings).
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/303/console
This message is automatically generated.
+1
I don't see the JavaDoc warning when I compile, so I don't know what Hudson saw...
Fixing the javac warnings (i think it was coming out of TestTokenStorage.java due to the use of a deprecated class). Also, changed the Interface audience for TokenStorage to be only MapReduce.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12431871/h-6520.4.patch
against trunk revision 904339.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/305/console
This message is automatically generated.
Integrated in Hadoop-Common-trunk-Commit #152 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/152/)
. Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.
. Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.
Integrated in Hadoop-Common-trunk #238 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/238/)
. Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.
. Adds APIs to read/write Token and secret keys. Also adds the automatic loading of tokens into UserGroupInformation upon login. The tokens are read from a file specified in the environment variable. Contributed by Devaraj Das.
This environment variable exists - JOB_TOKEN_FILE. In this jira, we should read the contents of the file via TokenCache's APIs and load the tokens in the UGI (similar to the way job-token is loaded in Child.java). In the JobInProgress, the tokens already exist in memory and it should be easy to add it to the ugi used for copying the jobconf file from the hdfs. In the tasktracker, ditto (for localizing the job's files).
For tasks (and for other processes like an MR Streaming app running a hdfs command from within the tasks), if we have a static block in the UGI to look for the env variable, and load the token cache using the TokenCache's APIs, it should work. Also, when the env var is defined, we should use simple authentication for login.
Thoughts?