Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-5535 Umbrella jira for improved HDFS rolling upgrades
  3. HDFS-6031

NN running newer software rejects loading the fsimage during rolling upgrade.

    Details

    • Hadoop Flags:
      Reviewed

      Description

      During rolling upgrade, the standby NN that has a newer layout version will complain about that the fsimage is too old, and require starting the NN using the -upgrade option:

      File system image contains an old layout version -55. An upgrade to version -56 is required.
      

      However, -upgrade is disabled during rolling upgrade, thus it is impossible to restart the standby NN:

      org.apache.hadoop.hdfs.protocol.RollingUpgradeException: Failed to upgrade namenode since a rolling upgrade is already in progress. Existing rolling upgrade info:
      ...
      
      1. h6031_20140227.patch
        0.7 kB
        Tsz Wo Nicholas Sze
      2. h6031_20140227b.patch
        2 kB
        Tsz Wo Nicholas Sze
      3. h6031_20140227c.patch
        6 kB
        Tsz Wo Nicholas Sze
      4. h6031_20140227d.patch
        13 kB
        Tsz Wo Nicholas Sze
      5. h6031_20140227e.patch
        13 kB
        Tsz Wo Nicholas Sze

        Issue Links

          Activity

          Hide
          Tsz Wo Nicholas Sze added a comment -

          Good catch! Since we have protobuf, the layout version won't be change in the near future. Anyway, let's fix it.

          h6031_20140227.patch

          Show
          Tsz Wo Nicholas Sze added a comment - Good catch! Since we have protobuf, the layout version won't be change in the near future. Anyway, let's fix it. h6031_20140227.patch
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The previous patch did not work since the rolling upgrade info is not loaded.

          h6031_20140227b.patch: adds back the "-rollingUpgrade started" startup option.

          Show
          Tsz Wo Nicholas Sze added a comment - The previous patch did not work since the rolling upgrade info is not loaded. h6031_20140227b.patch: adds back the "-rollingUpgrade started" startup option.
          Hide
          Haohui Mai added a comment -

          Looks good to me. I've verified that NN can start with -rollingUpgrade started.

          Nit: there are trailing white spaces in the patch. +1 once addressed.

          Show
          Haohui Mai added a comment - Looks good to me. I've verified that NN can start with -rollingUpgrade started . Nit: there are trailing white spaces in the patch. +1 once addressed.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h6031_20140227c.patch: also updates the doc.

          Show
          Tsz Wo Nicholas Sze added a comment - h6031_20140227c.patch: also updates the doc.
          Hide
          Haohui Mai added a comment -

          +1.

          Nit: there's one trailing white space in this line.

          +          && option.getRollingUpgradeStartupOption() 
          
          Show
          Haohui Mai added a comment - +1. Nit: there's one trailing white space in this line. + && option.getRollingUpgradeStartupOption()
          Hide
          Tsz Wo Nicholas Sze added a comment -

          h6031_20140227d.patch: do not verify layout version during rolling upgrade.

          Show
          Tsz Wo Nicholas Sze added a comment - h6031_20140227d.patch: do not verify layout version during rolling upgrade.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          > Nit: there are trailing white spaces in the patch. +1 once addressed.

          Sorry that I forgot to address this previously. Generally, we should not pay too much attention on the white spaces. It is nice to have but should not be a requirement. Let's put our energy on the code. Otherwise, it will slow down the development progress.

          h6031_20140227e.patch: removes the white space.

          Show
          Tsz Wo Nicholas Sze added a comment - > Nit: there are trailing white spaces in the patch. +1 once addressed. Sorry that I forgot to address this previously. Generally, we should not pay too much attention on the white spaces. It is nice to have but should not be a requirement. Let's put our energy on the code. Otherwise, it will slow down the development progress. h6031_20140227e.patch: removes the white space.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Let me commit h6031_20140227c.patch first and work on the remaining bug in HDFS-6034.

          Show
          Tsz Wo Nicholas Sze added a comment - Let me commit h6031_20140227c.patch first and work on the remaining bug in HDFS-6034 .
          Hide
          Tsz Wo Nicholas Sze added a comment -

          Thanks Haohui, for testing and reviewing the patch.

          I have committed this.

          Show
          Tsz Wo Nicholas Sze added a comment - Thanks Haohui, for testing and reviewing the patch. I have committed this.

            People

            • Assignee:
              Tsz Wo Nicholas Sze
              Reporter:
              Haohui Mai
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development