Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10719

In HA, Namenode is failed to start If any of the Quorum hostname is unresolved.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.7.1
    • Fix Version/s: None
    • Component/s: journal-node, namenode
    • Labels:
    • Environment:

      HDP-2.4

    • Flags:
      Patch

      Description

      2016-08-03 02:53:53,760 ERROR namenode.NameNode (NameNode.java:main(1712)) - Failed to start namenode.
      java.lang.IllegalArgumentException: Unable to construct journal, qjournal://xxxx1:8485;xxxx2:8485;xxxx3:8485/shva
      at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1637)
      at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournals(FSEditLog.java:282)
      at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initSharedJournalsForRead(FSEditLog.java:260)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.initEditLog(FSImage.java:789)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:634)
      at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:983)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:688)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:662)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:726)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:951)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:935)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1641)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1707)
      Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
      at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1635)
      ... 13 more
      Caused by: java.lang.NullPointerException
      at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannelMetrics.getName(IPCLoggerChannelMetrics.java:107)
      at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannelMetrics.create(IPCLoggerChannelMetrics.java:91)
      at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel.<init>(IPCLoggerChannel.java:178)
      at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$1.createLogger(IPCLoggerChannel.java:156)
      at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:367)
      at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:149)
      at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.<init>(QuorumJournalManager.java:116)
      at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.<init>(QuorumJournalManager.java:105)
      ... 18 more
      2016-08-03 02:53:53,765 INFO util.ExitUtil (ExitUtil.java:terminate(124)) - Exiting with status 1
      2016-08-03 02:53:53,768 INFO namenode.NameNode (LogAdapter.java:info(47)) - SHUTDOWN_MSG:

      and the failover is not successful

      I have attached the patch, It allows the Namenode to start if the majority of the Quorums are resolvable.
      throws warning if the quorum is unresolvable.
      throws Unknown host exception if the majority of the journals are unresolvable.

        Attachments

        1. HDFS-10719-1.patch
          2 kB
          Karthik Palanisamy
        2. HDFS-10719-2.patch
          2 kB
          Karthik Palanisamy
        3. HDFS-10719-3.patch
          2 kB
          Karthik Palanisamy
        4. HDFS-10719-4.patch
          1 kB
          Karthik Palanisamy

          Issue Links

            Activity

              People

              • Assignee:
                kpalanisamy Karthik Palanisamy
                Reporter:
                kpalanisamy Karthik Palanisamy
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: