[AMBARI-25438] Ambari starts NN with incorrect flag during upgrade

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7.3
• Component/s: ambari-server
• Assignee: Unassigned
• Reporter: amarnath reddy pappu
• Created: 02/Dec/19 22:39
• Updated: 02/Dec/19 22:42

Description

Steps to reproduce:

1. Install Ambari 2.6.x and HDP 2.6.x
2. enable NN HA
3. Upgrade Ambari 2.7.x
4. Now Start HDP upgrade to 3.x and after HDFS upgrade is done pause the Upgrade
5. Now stop both NNs are then attempt to start one from Ambari UI -  then Ambari would attempt to start the NN with "–upgrade flag"
6. If so it fails to come up with below exception.

2019-12-02 22:05:58,627 ERROR namenode.NameNode (NameNode.java:main(1715)) - Failed to start namenode. org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /hadoop/hdfs/namenode is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first. at org.apache.hadoop.hdfs.server.namenode.FSImage.checkUpgrade(FSImage.java:411) at org.apache.hadoop.hdfs.server.namenode.FSImage.checkUpgrade(FSImage.java:418) at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:438) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:310) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1090) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:694) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:937) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:910) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710) 2019-12-02 22:05:58,629 INFO util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 1: org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /hadoop/hdfs/namenode is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first.

Ambari should check if NN upgrade is already done - if it is done already then it should not check start "upgrade" flag.