Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11209

SNN can't checkpoint when rolling upgrade is not finalized

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.8.0, 3.0.0-alpha1
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: rolling upgrades
    • Labels:
      None

      Description

      Similar problem has been fixed with HDFS-7185. Recent change in HDFS-8432 brings this back.

      With HDFS-8432, the primary NN will not update the VERSION file to the new version after running with "rollingUpgrade" option until upgrade is finalized. This is to support more downgrade use cases.

      However, the checkpoint on the SNN is incorrectly updating the VERSION file when the rollingUpgrade is not finalized yet on the primary NN. As a result, the SNN checkpoint successfully but fail to push it to the primary NN because its version is higher than the primary NN as shown below.

      2016-12-02 05:25:31,918 ERROR namenode.SecondaryNameNode (SecondaryNameNode.java:doWork(399)) - Exception in doCheckpoint
      org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpPutFailedException: Image uploading failed, status: 403, url: http://NN:50070/imagetransfer?txid=345404754&imageFile=IMAGE&File-Le..., message: This namenode has storage info -60:221856466:1444080250181:clusterX but the secondary expected -63:221856466:1444080250181:clusterX
      

        Attachments

        1. HDFS-11209.04.patch
          10 kB
          Xiaoyu Yao
        2. HDFS-11209.03.patch
          11 kB
          Xiaoyu Yao
        3. HDFS-11209.02.patch
          10 kB
          Xiaoyu Yao
        4. HDFS-11209.01.patch
          9 kB
          Xiaoyu Yao
        5. HDFS-11209.00.patch
          7 kB
          Xiaoyu Yao

          Issue Links

            Activity

              People

              • Assignee:
                xyao Xiaoyu Yao
                Reporter:
                xyao Xiaoyu Yao
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: