Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.23.1, 2.0.0-alpha, 3.0.0
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: security
    • Labels:
      None

      Description

      When DistCp is invoked through a proxy-user (e.g. through Oozie), the delegation-token-store isn't picked up by DistCp correctly. One sees failures such as:

      ERROR [main] org.apache.hadoop.tools.DistCp: Couldn't complete DistCp
      operation:
      java.lang.SecurityException: Intercepted System.exit(-999)
      at
      org.apache.oozie.action.hadoop.LauncherSecurityManager.checkExit(LauncherMapper.java:651)
      at java.lang.Runtime.exit(Runtime.java:88)
      at java.lang.System.exit(System.java:904)
      at org.apache.hadoop.tools.DistCp.main(DistCp.java:357)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at
      sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at
      sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at
      org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:394)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:399)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:334)
      at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:147)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at
      org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
      at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:142)

      Looking over the DistCp code, one sees that HADOOP_TOKEN_FILE_LOCATION isn't being copied to mapreduce.job.credentials.binary, in the job-conf. I'll post a patch for this shortly.

      1. HADOOP-8225.patch
        20 kB
        Daryn Sharp
      2. HADOOP-8225.patch
        10 kB
        Daryn Sharp
      3. HADOOP-8225.patch
        0.9 kB
        Daryn Sharp
      4. HADOOP-8225.patch
        0.8 kB
        Thomas Graves
      5. HADOOP-8225.patch
        0.8 kB
        Mithun Radhakrishnan

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1183 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1183/)
          HDFS-3852. TestHftpDelegationToken is broken after HADOOP-8225 (daryn) (Revision 1379623)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379623
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1183 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1183/ ) HDFS-3852 . TestHftpDelegationToken is broken after HADOOP-8225 (daryn) (Revision 1379623) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379623 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1152 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1152/)
          HDFS-3852. TestHftpDelegationToken is broken after HADOOP-8225 (daryn) (Revision 1379623)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379623
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1152 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1152/ ) HDFS-3852 . TestHftpDelegationToken is broken after HADOOP-8225 (daryn) (Revision 1379623) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379623 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #361 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/361/)
          svn merge -c 1379623 FIXES: HDFS-3852. TestHftpDelegationToken is broken after HADOOP-8225 (daryn) (Revision 1379627)

          Result = UNSTABLE
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379627
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #361 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/361/ ) svn merge -c 1379623 FIXES: HDFS-3852 . TestHftpDelegationToken is broken after HADOOP-8225 (daryn) (Revision 1379627) Result = UNSTABLE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379627 Files : /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2693 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2693/)
          HDFS-3852. TestHftpDelegationToken is broken after HADOOP-8225 (daryn) (Revision 1379623)

          Result = FAILURE
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379623
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2693 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2693/ ) HDFS-3852 . TestHftpDelegationToken is broken after HADOOP-8225 (daryn) (Revision 1379623) Result = FAILURE daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379623 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2731 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2731/)
          HDFS-3852. TestHftpDelegationToken is broken after HADOOP-8225 (daryn) (Revision 1379623)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379623
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2731 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2731/ ) HDFS-3852 . TestHftpDelegationToken is broken after HADOOP-8225 (daryn) (Revision 1379623) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379623 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2668 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2668/)
          HDFS-3852. TestHftpDelegationToken is broken after HADOOP-8225 (daryn) (Revision 1379623)

          Result = SUCCESS
          daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379623
          Files :

          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
          • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2668 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2668/ ) HDFS-3852 . TestHftpDelegationToken is broken after HADOOP-8225 (daryn) (Revision 1379623) Result = SUCCESS daryn : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1379623 Files : /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpDelegationToken.java
          Hide
          Aaron T. Myers added a comment -

          Looks like this patch also broken TestHftpDelegationToken. I've filed HDFS-3852 to address that issue. Could somebody please take a look at that?

          Show
          Aaron T. Myers added a comment - Looks like this patch also broken TestHftpDelegationToken. I've filed HDFS-3852 to address that issue. Could somebody please take a look at that?
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1175 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1175/)
          HADOOP-8225. DistCp fails when invoked by Oozie (daryn via bobby) (Revision 1376618)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376618
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCredentials.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1175 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1175/ ) HADOOP-8225 . DistCp fails when invoked by Oozie (daryn via bobby) (Revision 1376618) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376618 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCredentials.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
          Hide
          Ivan Kelly added a comment -

          Great. Thanks Daryn

          Show
          Ivan Kelly added a comment - Great. Thanks Daryn
          Hide
          Daryn Sharp added a comment -

          Yes, I've already fixed. Please see the linked HADOOP-8725.

          Show
          Daryn Sharp added a comment - Yes, I've already fixed. Please see the linked HADOOP-8725 .
          Hide
          Ivan Kelly added a comment -

          I'm trying to run current hadoop trunk with hbase, and I've traced a problem to this change.

          When I run(in hbase-server + HBASE-6581's patch):
          mvn -Dhadoop.profile=3.0 test -PlocalTests -Dtest=TestHFileOutputFormat#testMRIncrementalLoad

          I get the following in the log.

          2012-08-24 11:54:50,528 WARN [main] org.apache.hadoop.security.SecurityUtil: Failed to get token for service 10.78.36.76:59635
          2012-08-24 11:54:50,532 ERROR [main] org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[main,5,main] threw an Exception.
          java.lang.NullPointerException
          at org.apache.hadoop.security.UserGroupInformation.addToken(UserGroupInformation.java:1222)
          at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:102)

          From digging to YarnChild, it seems that after this change, YarnChild never initializes the TokenCache. Is this correct? How is the TokenCache supposed to be initialized after this change?

          Show
          Ivan Kelly added a comment - I'm trying to run current hadoop trunk with hbase, and I've traced a problem to this change. When I run(in hbase-server + HBASE-6581 's patch): mvn -Dhadoop.profile=3.0 test -PlocalTests -Dtest=TestHFileOutputFormat#testMRIncrementalLoad I get the following in the log. 2012-08-24 11:54:50,528 WARN [main] org.apache.hadoop.security.SecurityUtil: Failed to get token for service 10.78.36.76:59635 2012-08-24 11:54:50,532 ERROR [main] org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread [main,5,main] threw an Exception. java.lang.NullPointerException at org.apache.hadoop.security.UserGroupInformation.addToken(UserGroupInformation.java:1222) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:102) From digging to YarnChild, it seems that after this change, YarnChild never initializes the TokenCache. Is this correct? How is the TokenCache supposed to be initialized after this change?
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1144 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1144/)
          HADOOP-8225. DistCp fails when invoked by Oozie (daryn via bobby) (Revision 1376618)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376618
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCredentials.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1144 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1144/ ) HADOOP-8225 . DistCp fails when invoked by Oozie (daryn via bobby) (Revision 1376618) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376618 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCredentials.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #353 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/353/)
          svn merge -c 1376618 FIXES: HADOOP-8225. DistCp fails when invoked by Oozie (daryn via bobby) (Revision 1376624)

          Result = UNSTABLE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376624
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCredentials.java
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
          • /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #353 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/353/ ) svn merge -c 1376618 FIXES: HADOOP-8225 . DistCp fails when invoked by Oozie (daryn via bobby) (Revision 1376624) Result = UNSTABLE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376624 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCredentials.java /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2656 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2656/)
          HADOOP-8225. DistCp fails when invoked by Oozie (daryn via bobby) (Revision 1376618)

          Result = FAILURE
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376618
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCredentials.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2656 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2656/ ) HADOOP-8225 . DistCp fails when invoked by Oozie (daryn via bobby) (Revision 1376618) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376618 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCredentials.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2628 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2628/)
          HADOOP-8225. DistCp fails when invoked by Oozie (daryn via bobby) (Revision 1376618)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376618
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCredentials.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2628 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2628/ ) HADOOP-8225 . DistCp fails when invoked by Oozie (daryn via bobby) (Revision 1376618) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376618 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCredentials.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2692 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2692/)
          HADOOP-8225. DistCp fails when invoked by Oozie (daryn via bobby) (Revision 1376618)

          Result = SUCCESS
          bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376618
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCredentials.java
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java
          • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java
          • /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2692 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2692/ ) HADOOP-8225 . DistCp fails when invoked by Oozie (daryn via bobby) (Revision 1376618) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1376618 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/Credentials.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestCredentials.java /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/TestUserGroupInformation.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapred/YarnChild.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/MRAppMaster.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Job.java /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/TestJob.java /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java
          Hide
          Robert Joseph Evans added a comment -

          Thanks Daryn,

          I put this into trunk, branch-2, branch-2.1.0-alpha, and branch-0.23

          Show
          Robert Joseph Evans added a comment - Thanks Daryn, I put this into trunk, branch-2, branch-2.1.0-alpha, and branch-0.23
          Hide
          Robert Joseph Evans added a comment -

          I have spent some time digging into this too and I feel confident that it works. +1 I'll check this in.

          Show
          Robert Joseph Evans added a comment - I have spent some time digging into this too and I feel confident that it works. +1 I'll check this in.
          Hide
          Devaraj Das added a comment -

          The patch looks good on the tokens side. +1.

          Show
          Devaraj Das added a comment - The patch looks good on the tokens side. +1.
          Hide
          Devaraj Das added a comment -

          Will take a look.

          Show
          Devaraj Das added a comment - Will take a look.
          Hide
          Robert Joseph Evans added a comment -

          I have done a quick look thorough the patch and it looks good to me. I am holding off on a full +1 until I have looked more deeply into the patch as I am not really a token expert. If someone else gives it a +1 feel free to check it in without waiting for me.

          Show
          Robert Joseph Evans added a comment - I have done a quick look thorough the patch and it looks good to me. I am holding off on a full +1 until I have looked more deeply into the patch as I am not really a token expert. If someone else gives it a +1 feel free to check it in without waiting for me.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12542139/HADOOP-8225.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-distcp.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1353//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1353//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542139/HADOOP-8225.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-distcp. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1353//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1353//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          Marking as blocker since distcp and other apps that launch jobs can't work under oozie.

          Show
          Daryn Sharp added a comment - Marking as blocker since distcp and other apps that launch jobs can't work under oozie.
          Hide
          Daryn Sharp added a comment -

          Add a bunch of tests. Corrected NamedToken's equality impl.

          Show
          Daryn Sharp added a comment - Add a bunch of tests. Corrected NamedToken 's equality impl.
          Hide
          Daryn Sharp added a comment -

          How about s/devious/diabolically clever/? I can't think of another way to do it either.

          Yes, currently writing tests.

          Show
          Daryn Sharp added a comment - How about s/devious/diabolically clever/? I can't think of another way to do it either. Yes, currently writing tests.
          Hide
          Alejandro Abdelnur added a comment -

          Daryn, are you planning to write a testcase to complete the patch?

          Show
          Alejandro Abdelnur added a comment - Daryn, are you planning to write a testcase to complete the patch?
          Hide
          Alejandro Abdelnur added a comment -

          Approach LGTM. Regarding the devious way for Oozie to trap System.exit() calls, it is the only way we found we could use tools that don't have a programmatic entry point but the main() method, and relay on System.exit() to indicate success/failure.

          Show
          Alejandro Abdelnur added a comment - Approach LGTM. Regarding the devious way for Oozie to trap System.exit() calls, it is the only way we found we could use tools that don't have a programmatic entry point but the main() method, and relay on System.exit() to indicate success/failure.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12542025/HADOOP-8225.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          -1 eclipse:eclipse. The patch failed to build with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-distcp:

          org.apache.hadoop.security.TestUserGroupInformation
          org.apache.hadoop.ha.TestZKFailoverController

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1347//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1347//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12542025/HADOOP-8225.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. -1 eclipse:eclipse. The patch failed to build with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core hadoop-tools/hadoop-distcp: org.apache.hadoop.security.TestUserGroupInformation org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1347//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1347//console This message is automatically generated.
          Hide
          Devaraj Das added a comment -

          This approach feels right.

          Show
          Devaraj Das added a comment - This approach feels right.
          Hide
          Daryn Sharp added a comment -

          It has been tested on a live cluster though.

          Show
          Daryn Sharp added a comment - It has been tested on a live cluster though.
          Hide
          Daryn Sharp added a comment -

          No tests written, pending approval of approach

          Show
          Daryn Sharp added a comment - No tests written, pending approval of approach
          Hide
          Daryn Sharp added a comment -

          This is actually caused by multiple bugs:

          1. MR job submission requests tokens it already has
          2. MR job submission doesn't always pass all tokens (JHS, MR, HIVE, etc)
          3. Oozie is using a devious way to detect the exit code of an action

          Details:

          1. The reported exception occurs when a task tries to get tokens it ALREADY has. Job submission gets missing tokens for input/output paths and adds them to the UGI for RPC connections. Job submission doesn't check the UGI, so it doesn't think it has the token, thus requests another. The NN connection uses the token that the job doesn't think it has! The NN squawks that you can't use a token to get a token.
          2. Similarly, distcp also does some prep work to acquire tokens prior to job submission. So again, a task tries and fails to get the tokens it already has.... Invoking a command like distcp directly will "work" (masks the bug) because it uses the TGT to get another token even if it already has one in the UGI.
          3. Job submission doesn't appear to propagate non-FS/MR tokens in the task's UGI into the new job submission.
          4. Oozie uses a security manager to intercept an action's System.exit, throws a SecurityException containing the exit code, and later catches that exception to determine success/failure. Devious! Distcp calls System.exit(0) inside a try block which catches oozie's SecurityException, logs it, and then calls System.exit(-999), again generating an oozie SecurityException. Due to the try/catch, distcp will ALWAYS appear to fail.

          Solutions:

          • #1,2,3: Seeding Job with existing UGI tokens
          • #4 Distcp calls System.exit OUTSIDE of the try block

          I seeded the Job's credentials with the existing UGI tokens because it seems unreasonable to require all apps that launch jobs to be aware of running as a task.

          Show
          Daryn Sharp added a comment - This is actually caused by multiple bugs: MR job submission requests tokens it already has MR job submission doesn't always pass all tokens (JHS, MR, HIVE, etc) Oozie is using a devious way to detect the exit code of an action Details: The reported exception occurs when a task tries to get tokens it ALREADY has. Job submission gets missing tokens for input/output paths and adds them to the UGI for RPC connections. Job submission doesn't check the UGI, so it doesn't think it has the token, thus requests another. The NN connection uses the token that the job doesn't think it has! The NN squawks that you can't use a token to get a token. Similarly, distcp also does some prep work to acquire tokens prior to job submission. So again, a task tries and fails to get the tokens it already has.... Invoking a command like distcp directly will "work" (masks the bug) because it uses the TGT to get another token even if it already has one in the UGI. Job submission doesn't appear to propagate non-FS/MR tokens in the task's UGI into the new job submission. Oozie uses a security manager to intercept an action's System.exit, throws a SecurityException containing the exit code, and later catches that exception to determine success/failure. Devious! Distcp calls System.exit(0) inside a try block which catches oozie's SecurityException, logs it, and then calls System.exit(-999), again generating an oozie SecurityException. Due to the try/catch, distcp will ALWAYS appear to fail. Solutions: #1,2,3: Seeding Job with existing UGI tokens #4 Distcp calls System.exit OUTSIDE of the try block I seeded the Job's credentials with the existing UGI tokens because it seems unreasonable to require all apps that launch jobs to be aware of running as a task.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12538787/HADOOP-8225.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1242//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1242//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538787/HADOOP-8225.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1242//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1242//console This message is automatically generated.
          Hide
          Daryn Sharp added a comment -

          I think this one-liner should "fix" the problem, but it's completely untested. I think the whole token file handling needs to analyzed and cleaned up on a separate jira.

          Show
          Daryn Sharp added a comment - I think this one-liner should "fix" the problem, but it's completely untested. I think the whole token file handling needs to analyzed and cleaned up on a separate jira.
          Hide
          Daryn Sharp added a comment -

          Actually, the fact that multiple projects need to make a similar change is indicative of a bug in the core. The passing along of tokens should be an invisible service provide by the hadoop security framework. Distcp and other tools should not know or care whether it's being run via a real user, proxy user, via oozie, or something else.

          I briefly investigated a bit more, and the env var actually originates from UGI which internally uses it to automatically load the tokens, hence tools should not know about this internal implementation detail of UGI. I think the issue is the tokens are being loaded into the login user instead of the real user. I believe no tool will require changes if the UGI is modified to load the tokens into the real user.

          Show
          Daryn Sharp added a comment - Actually, the fact that multiple projects need to make a similar change is indicative of a bug in the core. The passing along of tokens should be an invisible service provide by the hadoop security framework. Distcp and other tools should not know or care whether it's being run via a real user, proxy user, via oozie, or something else. I briefly investigated a bit more, and the env var actually originates from UGI which internally uses it to automatically load the tokens, hence tools should not know about this internal implementation detail of UGI. I think the issue is the tokens are being loaded into the login user instead of the real user. I believe no tool will require changes if the UGI is modified to load the tokens into the real user.
          Hide
          Mithun Radhakrishnan added a comment -

          Hello, Daryn. Thanks so much for reviewing the fix.

          I'm pretty sure that DistCp handles tokens correctly. This code-path was introduced purely for the case where DistCp is invoked via Oozie. I wish there were another way to transfer delegation-tokens from Oozie's launcher over to DistCp. (This is also the way Pig and Hive actions work in Oozie.)

          Show
          Mithun Radhakrishnan added a comment - Hello, Daryn. Thanks so much for reviewing the fix. I'm pretty sure that DistCp handles tokens correctly. This code-path was introduced purely for the case where DistCp is invoked via Oozie. I wish there were another way to transfer delegation-tokens from Oozie's launcher over to DistCp. (This is also the way Pig and Hive actions work in Oozie.)
          Hide
          Daryn Sharp added a comment -

          Yarn tasks are supposed to load the credentials into the conf, and then job submission is supposed to write them back out from the conf. It's supposed to be transparent. This patch is assuming private knowledge of the aforementioned inner workings by hardcoding a yarn env var (ApplicationConstants.CONTAINER_TOKEN_FILE_ENV_NAME), and hardcoding a mapreduce conf key (MAPREDUCE_JOB_CREDENTIALS_BINARY).

          Are you sure the conf isn't being mishandled which is losing the tokens?

          Show
          Daryn Sharp added a comment - Yarn tasks are supposed to load the credentials into the conf, and then job submission is supposed to write them back out from the conf. It's supposed to be transparent. This patch is assuming private knowledge of the aforementioned inner workings by hardcoding a yarn env var ( ApplicationConstants.CONTAINER_TOKEN_FILE_ENV_NAME ), and hardcoding a mapreduce conf key ( MAPREDUCE_JOB_CREDENTIALS_BINARY ). Are you sure the conf isn't being mishandled which is losing the tokens?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12538199/HADOOP-8225.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-distcp.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1225//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1225//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538199/HADOOP-8225.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-tools/hadoop-distcp. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1225//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1225//console This message is automatically generated.
          Hide
          Thomas Graves added a comment -

          re-attaching same patch to kick jenkins.

          Show
          Thomas Graves added a comment - re-attaching same patch to kick jenkins.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12520480/HADOOP-8225.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/797//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520480/HADOOP-8225.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/797//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12520465/HADOOP-8225-branch-0.23.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/796//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12520465/HADOOP-8225-branch-0.23.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/796//console This message is automatically generated.
          Hide
          Mithun Radhakrishnan added a comment -

          Renamed patches, for Hadoop common.

          Show
          Mithun Radhakrishnan added a comment - Renamed patches, for Hadoop common.
          Hide
          Dave Thompson added a comment -

          Hey Mithun, your patch applies to both 0.23.1 and trunk for me as well. I believe hudson is failing to apply the patch because it's a "MAPREDUCE-" ticket which doesn't touch the hadoop-tools/* directory. I believe you will need to move this to Hadoop common for Hudson's sake.

          Show
          Dave Thompson added a comment - Hey Mithun, your patch applies to both 0.23.1 and trunk for me as well. I believe hudson is failing to apply the patch because it's a "MAPREDUCE-" ticket which doesn't touch the hadoop-tools/* directory. I believe you will need to move this to Hadoop common for Hudson's sake.
          Hide
          Mithun Radhakrishnan added a comment -

          Alright, I'm at a loss. I don't see why this patch doesn't apply to trunk/ or branch-0.23.1. (I've verified that it works manually.)

          Show
          Mithun Radhakrishnan added a comment - Alright, I'm at a loss. I don't see why this patch doesn't apply to trunk/ or branch-0.23.1. (I've verified that it works manually.)
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12519313/MAPREDUCE-4004-branch-0.23.1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2081//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12519313/MAPREDUCE-4004-branch-0.23.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2081//console This message is automatically generated.
          Hide
          Mithun Radhakrishnan added a comment -

          Renamed patches. Should apply to trunk and branches, identically.

          Show
          Mithun Radhakrishnan added a comment - Renamed patches. Should apply to trunk and branches, identically.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12518235/DistCp_HadoopTokenFileLocation.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2045//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12518235/DistCp_HadoopTokenFileLocation.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2045//console This message is automatically generated.
          Hide
          Mithun Radhakrishnan added a comment -

          Patch available for 0.23.1/ branch. Should apply as is to trunk/ as well.

          Show
          Mithun Radhakrishnan added a comment - Patch available for 0.23.1/ branch. Should apply as is to trunk/ as well.

            People

            • Assignee:
              Daryn Sharp
              Reporter:
              Mithun Radhakrishnan
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development