Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15059

3.0 deployment cannot work with old version MR tar ball which breaks rolling upgrade

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • None
    • 3.0.0
    • security
    • None
    • Reviewed
    • Hide
      This change reverses the default delegation token format implemented by HADOOP-12563, but preserves the capability to read the new delegation token format. When the new format becomes default, then MR deployment jobs runs will be compatible with releases that contain this change.
      Show
      This change reverses the default delegation token format implemented by HADOOP-12563 , but preserves the capability to read the new delegation token format. When the new format becomes default, then MR deployment jobs runs will be compatible with releases that contain this change.

    Description

      I tried to deploy 3.0 cluster with 2.9 MR tar ball. The MR job is failed because following error:

      2017-11-21 12:42:50,911 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1511295641738_0003_000001
      2017-11-21 12:42:51,070 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      2017-11-21 12:42:51,118 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
      java.lang.RuntimeException: Unable to determine current user
      	at org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:254)
      	at org.apache.hadoop.conf.Configuration$Resource.<init>(Configuration.java:220)
      	at org.apache.hadoop.conf.Configuration$Resource.<init>(Configuration.java:212)
      	at org.apache.hadoop.conf.Configuration.addResource(Configuration.java:888)
      	at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1638)
      Caused by: java.io.IOException: Exception reading /tmp/nm-local-dir/usercache/jdu/appcache/application_1511295641738_0003/container_e03_1511295641738_0003_01_000001/container_tokens
      	at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:208)
      	at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:907)
      	at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:820)
      	at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:689)
      	at org.apache.hadoop.conf.Configuration$Resource.getRestrictParserDefault(Configuration.java:252)
      	... 4 more
      Caused by: java.io.IOException: Unknown version 1 in token storage.
      	at org.apache.hadoop.security.Credentials.readTokenStorageStream(Credentials.java:226)
      	at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:205)
      	... 8 more
      2017-11-21 12:42:51,122 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1: java.lang.RuntimeException: Unable to determine current user
      

      I think it is due to token incompatiblity change between 2.9 and 3.0. As we claim "rolling upgrade" is supported in Hadoop 3, we should fix this before we ship 3.0 otherwise all MR running applications will get stuck during/after upgrade.

      Attachments

        1. HADOOP-15059.006.patch
          11 kB
          Vinod Kumar Vavilapalli
        2. HADOOP-15059.005.patch
          11 kB
          Jason Darrell Lowe
        3. HADOOP-15059.004.patch
          10 kB
          Jason Darrell Lowe
        4. HADOOP-15059.003.patch
          30 kB
          Jason Darrell Lowe
        5. HADOOP-15059.002.patch
          30 kB
          Jason Darrell Lowe
        6. HADOOP-15059.001.patch
          30 kB
          Jason Darrell Lowe

        Issue Links

          Activity

            People

              jlowe Jason Darrell Lowe
              junping_du Junping Du
              Votes:
              0 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: