Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-9096

Issue in Rollback (after rolling upgrade) from hadoop 2.7.1 to 2.4.0

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.4.0
    • Fix Version/s: None
    • Component/s: rolling upgrades
    • Labels:
      None
    • Flags:
      Important

      Description

      I tried to do rolling upgrade from hadoop 2.4.0 to hadoop 2.7.1. As per http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#dfsadmin_-rollingUpgrade one can rollback to previous release provided the finalise step is not done. I upgraded the setup but didnot finalise the upgrade and tried to rollback HDFS to 2.4.0

      I tried the following steps

      1. Shutdown all NNs and DNs.
      2. Restore the pre-upgrade release in all machines.
      3. Start NN1 as Active with the "-rollingUpgrade rollback<http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#namenode_-rollingUpgrade>" option.

      I am getting the following error after 3rd step

      15/09/01 17:53:35 INFO namenode.AclConfigFlag: ACLs enabled? false

      15/09/01 17:53:35 INFO common.Storage: Lock on <<NameNode dir>>/in_use.lock acquired by nodename 12152@VM-2

      15/09/01 17:53:35 WARN namenode.FSNamesystem: Encountered exception loading fsimage

      org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage directory /data/yarn/namenode. Reported: -63. Expecting = -56.

      at org.apache.hadoop.hdfs.server.common.StorageInfo.setLayoutVersion(StorageInfo.java:178)

      at org.apache.hadoop.hdfs.server.common.StorageInfo.setFieldsFromProperties(StorageInfo.java:131)

      at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:608)

      at org.apache.hadoop.hdfs.server.common.StorageInfo.readProperties(StorageInfo.java:228)

      at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:309)

      at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)

      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:882)

      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:639)

      at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:455)

      at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:511)

      at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:670)

      at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:655)

      at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1304)

      at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1370)

      15/09/01 17:53:35 INFO mortbay.log: Stopped SelectChannelConnector@0.0.0.0:50070

      15/09/01 17:53:35 INFO impl.MetricsSystemImpl: Stopping NameNode metrics system...

      15/09/01 17:53:35 INFO impl.MetricsSystemImpl: NameNode metrics system stopped.

      15/09/01 17:53:35 INFO impl.MetricsSystemImpl: NameNode metrics system shutdown complete.

      15/09/01 17:53:35 FATAL namenode.NameNode: Exception in namenode join

      From rolling upgrade documentation it can be inferred that rolling upgrade is supported Hadoop 2.4.0 onwards but rollingUpgrade rollback to Hadoop 2.4.0 seems to be broken in Hadoop 2.4.0. It throws above mentioned error.

      Are there any other steps to perform rollback (from rolling upgrade) or is it not supported to rollback to Hadoop 2.4.0.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              harpreet.kaur Harpreet Kaur
            • Votes:
              5 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated: