Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11209

SNN can't checkpoint when rolling upgrade is not finalized

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.8.0, 3.0.0-alpha1
    • 2.9.0, 3.0.0-alpha2
    • rolling upgrades
    • None

    Description

      Similar problem has been fixed with HDFS-7185. Recent change in HDFS-8432 brings this back.

      With HDFS-8432, the primary NN will not update the VERSION file to the new version after running with "rollingUpgrade" option until upgrade is finalized. This is to support more downgrade use cases.

      However, the checkpoint on the SNN is incorrectly updating the VERSION file when the rollingUpgrade is not finalized yet on the primary NN. As a result, the SNN checkpoint successfully but fail to push it to the primary NN because its version is higher than the primary NN as shown below.

      2016-12-02 05:25:31,918 ERROR namenode.SecondaryNameNode (SecondaryNameNode.java:doWork(399)) - Exception in doCheckpoint
      org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpPutFailedException: Image uploading failed, status: 403, url: http://NN:50070/imagetransfer?txid=345404754&imageFile=IMAGE&File-Le..., message: This namenode has storage info -60:221856466:1444080250181:clusterX but the secondary expected -63:221856466:1444080250181:clusterX
      

      Attachments

        1. HDFS-11209.04.patch
          10 kB
          Xiaoyu Yao
        2. HDFS-11209.03.patch
          11 kB
          Xiaoyu Yao
        3. HDFS-11209.02.patch
          10 kB
          Xiaoyu Yao
        4. HDFS-11209.01.patch
          9 kB
          Xiaoyu Yao
        5. HDFS-11209.00.patch
          7 kB
          Xiaoyu Yao

        Issue Links

          Activity

            People

              xyao Xiaoyu Yao
              xyao Xiaoyu Yao
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: