[HDFS-10719] In HA, Namenode is failed to start If any of the Quorum hostname is unresolved.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.7.1
• Component/s: journal-node, namenode
• Labels: patch

Environment

HDP-2.4

• Flags: Patch
• Assignee: Karthik Palanisamy
• Reporter: Karthik Palanisamy
• Created: 03/Aug/16 15:32
• Updated: 25/Jul/22 08:06
• Resolved: 31/Aug/16 12:59

Description

2016-08-03 02:53:53,760 ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode.
java.lang.IllegalArgumentException: Unable to construct journal, qjournal://xxxx1:8485;xxxx2:8485;xxxx3:8485/shva
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1637)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournals(FSEditLog.java:282)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initSharedJournalsForRead(FSEditLog.java:260)
at org.apache.hadoop.hdfs.server.namenode.FSImage.initEditLog(FSImage.java:789)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:634)
at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:983)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:688)
at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:662)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:726)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:951)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:935)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1641)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1707)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1635)
... 13 more
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannelMetrics.getName(IPCLoggerChannelMetrics.java:107)
at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannelMetrics.create(IPCLoggerChannelMetrics.java:91)
at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel.<init>(IPCLoggerChannel.java:178)
at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$1.createLogger(IPCLoggerChannel.java:156)
at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:367)
at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:149)
at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.<init>(QuorumJournalManager.java:116)
at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.<init>(QuorumJournalManager.java:105)
... 18 more
2016-08-03 02:53:53,765 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1
2016-08-03 02:53:53,768 INFO namenode.NameNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG:

and the failover is not successful

I have attached the patch, It allows the Namenode to start if the majority of the Quorums are resolvable.
throws warning if the quorum is unresolvable.
throws Unknown host exception if the majority of the journals are unresolvable.

Issue Links

• is duplicated by: HDFS-4210 Throw helpful exception when DNS entry for JournalNode cannot be resolved