Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6135

In HDFS upgrade with HA setup, JournalNode cannot handle layout version bump when rolling back

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha1
    • Fix Version/s: 2.4.0
    • Component/s: journal-node
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      While doing HDFS upgrade with HA setup, if the layoutversion gets changed in the upgrade, the rollback may trigger the following exception in JournalNodes (suppose the new software bumped the layoutversion from -55 to -56):

      14/03/21 01:01:53 FATAL namenode.NameNode: Exception in namenode join
      org.apache.hadoop.hdfs.qjournal.client.QuorumException: Could not check if roll back possible for one or more JournalNodes. 1 exceptions thrown:
      Unexpected version of storage directory /grid/1/tmp/journal/mycluster. Reported: -56. Expecting = -55.
      	at org.apache.hadoop.hdfs.server.common.StorageInfo.setLayoutVersion(StorageInfo.java:203)
      	at org.apache.hadoop.hdfs.server.common.StorageInfo.setFieldsFromProperties(StorageInfo.java:156)
      	at org.apache.hadoop.hdfs.server.common.StorageInfo.readProperties(StorageInfo.java:135)
      	at org.apache.hadoop.hdfs.qjournal.server.JNStorage.analyzeStorage(JNStorage.java:202)
      	at org.apache.hadoop.hdfs.qjournal.server.JNStorage.<init>(JNStorage.java:73)
      	at org.apache.hadoop.hdfs.qjournal.server.Journal.<init>(Journal.java:142)
      	at org.apache.hadoop.hdfs.qjournal.server.JournalNode.getOrCreateJournal(JournalNode.java:87)
      	at org.apache.hadoop.hdfs.qjournal.server.JournalNode.canRollBack(JournalNode.java:304)
      	at org.apache.hadoop.hdfs.qjournal.server.JournalNodeRpcServer.canRollBack(JournalNodeRpcServer.java:228)
      

      Looks like for rollback JN with old software cannot handle future layoutversion brought by new software.

        Attachments

        1. HDFS-6135.002.patch
          3 kB
          Jing Zhao
        2. HDFS-6135.001.patch
          17 kB
          Jing Zhao
        3. HDFS-6135.000.patch
          14 kB
          Jing Zhao
        4. HDFS-6135.test.txt
          5 kB
          Jing Zhao

          Activity

            People

            • Assignee:
              jingzhao Jing Zhao
              Reporter:
              jingzhao Jing Zhao
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: