Hadoop Common
  1. Hadoop Common
  2. HADOOP-9374

Add tokens from -tokenCacheFile into UGI

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha, 3.0.0, 0.23.7
    • Fix Version/s: 0.23.7, 2.0.4-alpha
    • Component/s: security
    • Labels:
      None

      Description

      GenericOptionsParser accepts a -tokenCacheFile option. However, it only sets the mapreduce.job.credentials.json conf value instead of also adding the tokens to the UGI so they are usable by the command being executed.

        Issue Links

          Activity

          Daryn Sharp created issue -
          Hide
          Daryn Sharp added a comment -

          I have long used this patch for token testing.

          Show
          Daryn Sharp added a comment - I have long used this patch for token testing.
          Daryn Sharp made changes -
          Field Original Value New Value
          Attachment HADOOP-9374.patch [ 12572393 ]
          Daryn Sharp made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12572393/HADOOP-9374.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 tests included appear to have a timeout.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.util.TestWinUtils

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2280//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2280//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572393/HADOOP-9374.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 tests included appear to have a timeout. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.util.TestWinUtils +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2280//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2280//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          Windows test failures are completely unrelated.

          Show
          Daryn Sharp added a comment - Windows test failures are completely unrelated.
          Hide
          Arpit Agarwal added a comment -

          TestWinUtils failure is tracked by HADOOP-9372.

          Show
          Arpit Agarwal added a comment - TestWinUtils failure is tracked by HADOOP-9372 .
          Hide
          Kihwal Lee added a comment -

          +1 It supersedes the existing feature of setting "mapreduce.job.credentials.json", since JobContext.ugi will be the same current user ugi which the option parser adds the tokens to, at least for the CLIs we have today.

          Show
          Kihwal Lee added a comment - +1 It supersedes the existing feature of setting "mapreduce.job.credentials.json", since JobContext.ugi will be the same current user ugi which the option parser adds the tokens to, at least for the CLIs we have today.
          Hide
          Daryn Sharp added a comment -

          Thanks Kihwal. I committed to branch trunk/2/23

          Show
          Daryn Sharp added a comment - Thanks Kihwal. I committed to branch trunk/2/23
          Daryn Sharp made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 3.0.0 [ 12320357 ]
          Fix Version/s 0.23.7 [ 12323956 ]
          Fix Version/s 2.0.4-alpha [ 12324135 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3434 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3434/)
          HADOOP-9374. Add tokens from -tokenCacheFile into UGI (daryn) (Revision 1454019)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454019
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGenericOptionsParser.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3434 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3434/ ) HADOOP-9374 . Add tokens from -tokenCacheFile into UGI (daryn) (Revision 1454019) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454019 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGenericOptionsParser.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #149 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/149/)
          HADOOP-9374. Add tokens from -tokenCacheFile into UGI (daryn) (Revision 1454019)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454019
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGenericOptionsParser.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #149 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/149/ ) HADOOP-9374 . Add tokens from -tokenCacheFile into UGI (daryn) (Revision 1454019) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454019 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGenericOptionsParser.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #547 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/547/)
          svn merge -c 1454019 FIXES: HADOOP-9374. Add tokens from -tokenCacheFile into UGI (daryn) (Revision 1454028)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454028
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGenericOptionsParser.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #547 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/547/ ) svn merge -c 1454019 FIXES: HADOOP-9374 . Add tokens from -tokenCacheFile into UGI (daryn) (Revision 1454028) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454028 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGenericOptionsParser.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1338 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1338/)
          HADOOP-9374. Add tokens from -tokenCacheFile into UGI (daryn) (Revision 1454019)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454019
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGenericOptionsParser.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1338 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1338/ ) HADOOP-9374 . Add tokens from -tokenCacheFile into UGI (daryn) (Revision 1454019) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454019 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGenericOptionsParser.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1366 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1366/)
          HADOOP-9374. Add tokens from -tokenCacheFile into UGI (daryn) (Revision 1454019)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454019
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGenericOptionsParser.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1366 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1366/ ) HADOOP-9374 . Add tokens from -tokenCacheFile into UGI (daryn) (Revision 1454019) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454019 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGenericOptionsParser.java
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Allen Wittenauer made changes -
          Fix Version/s 3.0.0 [ 12320357 ]
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #676 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/676/)
          HADOOP-9989. Bug introduced in HADOOP-9374, which parses the -tokenCacheFile as binary file but set it to the configuration as JSON file. (zxu via tucu) (tucu: rev b100949404843ed245ef4e118291f55b3fdc81b8)

          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGenericOptionsParser.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #676 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/676/ ) HADOOP-9989 . Bug introduced in HADOOP-9374 , which parses the -tokenCacheFile as binary file but set it to the configuration as JSON file. (zxu via tucu) (tucu: rev b100949404843ed245ef4e118291f55b3fdc81b8) hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGenericOptionsParser.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #1867 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1867/)
          HADOOP-9989. Bug introduced in HADOOP-9374, which parses the -tokenCacheFile as binary file but set it to the configuration as JSON file. (zxu via tucu) (tucu: rev b100949404843ed245ef4e118291f55b3fdc81b8)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGenericOptionsParser.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1867 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1867/ ) HADOOP-9989 . Bug introduced in HADOOP-9374 , which parses the -tokenCacheFile as binary file but set it to the configuration as JSON file. (zxu via tucu) (tucu: rev b100949404843ed245ef4e118291f55b3fdc81b8) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGenericOptionsParser.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1892 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1892/)
          HADOOP-9989. Bug introduced in HADOOP-9374, which parses the -tokenCacheFile as binary file but set it to the configuration as JSON file. (zxu via tucu) (tucu: rev b100949404843ed245ef4e118291f55b3fdc81b8)

          • hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java
          • hadoop-common-project/hadoop-common/CHANGES.txt
          • hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGenericOptionsParser.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1892 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1892/ ) HADOOP-9989 . Bug introduced in HADOOP-9374 , which parses the -tokenCacheFile as binary file but set it to the configuration as JSON file. (zxu via tucu) (tucu: rev b100949404843ed245ef4e118291f55b3fdc81b8) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java hadoop-common-project/hadoop-common/CHANGES.txt hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/util/TestGenericOptionsParser.java
          Kiran Kumar M R made changes -
          Link This issue relates to HADOOP-9989 [ HADOOP-9989 ]

            People

            • Assignee:
              Daryn Sharp
              Reporter:
              Daryn Sharp
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development