Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15059

3.0 deployment cannot work with old version MR tar ball which breaks rolling upgrade

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.0.0
    • Component/s: security
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Hide
      This change reverses the default delegation token format implemented by HADOOP-12563, but preserves the capability to read the new delegation token format. When the new format becomes default, then MR deployment jobs runs will be compatible with releases that contain this change.
      Show
      This change reverses the default delegation token format implemented by HADOOP-12563 , but preserves the capability to read the new delegation token format. When the new format becomes default, then MR deployment jobs runs will be compatible with releases that contain this change.

      Description

      I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed because following error:

      2017-11-21 12:42:50,911 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1511295641738_0003_000001
      2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      2017-11-21 12:42:51,118 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
      java.lang.RuntimeException: Unable to determine current user
      	at org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
      	at org.apache.hadoop.conf.Configuration$Resource.<init>(Configuration.java:220)
      	at org.apache.hadoop.conf.Configuration$Resource.<init>(Configuration.java:212)
      	at org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
      Caused by: java.io.IOException: Exception reading /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_000001/container_tokens
      	at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
      	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
      	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
      	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
      	at org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
      	... 4 more
      Caused by: java.io.IOException: Unknown version 1 in token storage.
      	at org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
      	at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
      	... 8 more
      2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1: java.lang.RuntimeException: Unable to determine current user
      

      I think it is due to token incompatiblity change between 2.9 and 3.0. As we claim "rolling upgrade" is supported in Hadoop 3, we should fix this before we ship 3.0 otherwise all MR running applications will get stuck during/after upgrade.

        Attachments

        1. HADOOP-15059.001.patch
          30 kB
          Jason Lowe
        2. HADOOP-15059.002.patch
          30 kB
          Jason Lowe
        3. HADOOP-15059.003.patch
          30 kB
          Jason Lowe
        4. HADOOP-15059.004.patch
          10 kB
          Jason Lowe
        5. HADOOP-15059.005.patch
          11 kB
          Jason Lowe
        6. HADOOP-15059.006.patch
          11 kB
          Vinod Kumar Vavilapalli

          Issue Links

            Activity

              People

              • Assignee:
                jlowe Jason Lowe
                Reporter:
                djp Junping Du
              • Votes:
                0 Vote for this issue
                Watchers:
                18 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: