[HDFS-11209] SNN can't checkpoint when rolling upgrade is not finalized

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.8.0, 3.0.0-alpha1
• Fix Version/s: 2.9.0, 3.0.0-alpha2
• Component/s: rolling upgrades
• Target Version/s: 2.9.0, 3.0.0-alpha2
• Hadoop Flags: Reviewed
• Assignee: Xiaoyu Yao
• Reporter: Xiaoyu Yao
• Created: 06/Dec/16 03:02
• Updated: 25/Oct/19 23:26
• Resolved: 18/Jan/17 01:46

Description

Similar problem has been fixed with HDFS-7185. Recent change in HDFS-8432 brings this back.

With HDFS-8432, the primary NN will not update the VERSION file to the new version after running with "rollingUpgrade" option until upgrade is finalized. This is to support more downgrade use cases.

However, the checkpoint on the SNN is incorrectly updating the VERSION file when the rollingUpgrade is not finalized yet on the primary NN. As a result, the SNN checkpoint successfully but fail to push it to the primary NN because its version is higher than the primary NN as shown below.

2016-12-02 05:25:31,918 ERROR namenode.SecondaryNameNode (SecondaryNameNode.java:doWork(399)) - Exception in doCheckpoint
org.apache.hadoop.hdfs.server.namenode.TransferFsImage$HttpPutFailedException: Image uploading failed, status: 403, url: http://NN:50070/imagetransfer?txid=345404754&imageFile=IMAGE&File-Le..., message: This namenode has storage info -60:221856466:1444080250181:clusterX but the secondary expected -63:221856466:1444080250181:clusterX

Issue Links

• is broken by: HDFS-8432 Introduce a minimum compatible layout version to allow downgrade in more rolling upgrade use cases.