[HDFS-17167] Observer NameNode -observer startup option conflicts with -rollingUpgrade startup option

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.3.4
• Fix Version/s: 3.4.0
• Component/s: namenode
• Labels: pull-requests-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Danny Becker
• Reporter: Danny Becker
• Created: 25/Aug/23 18:39
• Updated: 28/Jan/24 01:44
• Resolved: 06/Sep/23 16:17

Description

NameNode currently uses a "StartupOption" to decide whether to start a node as an Observer NameNode. This causes an issue during a rolling upgrade because "rollingUpgrade" is also a StartupOption and NameNode will only allow 1 startup option, choosing the last startup option in the list. Observer in our environment starts with the following startup options: ["-rollingUpgrade", "started", "-observer"]. This means that the rolling upgrade gets ignored which causes Observer to have an issue when an actual rolling upgrade is ongoing:

2023-08-23T14:59:03.486-0700,WARN,[main],org.apache.hadoop.hdfs.server.namenode.FSNamesystem,"Encountered exception loading fsimage
java.io.IOException: 
File system image contains an old layout version -63.
An upgrade to version -66 is required.
Please restart NameNode with the \'-rollingUpgrade started\' option if a rolling upgrade is already started; or restart NameNode with the \'-upgrade\' option to start a new upgrade.
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:271)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1116)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:724)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:681)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:763)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1013)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:992)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1743)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1811)
"

Issue Links

• links to: GitHub PR