Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4210

Throw helpful exception when DNS entry for JournalNode cannot be resolved

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Setting :
      qjournal://cdh4master01:8485;cdh4master02:8485;cdh4worker03:8485/hdfscluster
      cdh4master01 and cdh4master02 JournalNode up and running,
      cdh4worker03 not yet provisionning (no DNS entrie)

      With :
      `hadoop namenode -format` fails with :
      12/11/19 14:42:42 FATAL namenode.NameNode: Exception in namenode join
      java.lang.IllegalArgumentException: Unable to construct journal, qjournal://cdh4master01:8485;cdh4master02:8485;cdh4worker03:8485/hdfscluster
      at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1235)
      at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournals(FSEditLog.java:226)
      at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournalsForWrite(FSEditLog.java:193)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:745)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1099)
      at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)
      Caused by: java.lang.reflect.InvocationTargetException
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
      at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1233)
      ... 5 more
      Caused by: java.lang.NullPointerException
      at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannelMetrics.getName(IPCLoggerChannelMetrics.java:107)
      at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannelMetrics.create(IPCLoggerChannelMetrics.java:91)
      at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel.<init>(IPCLoggerChannel.java:161)
      at org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel$1.createLogger(IPCLoggerChannel.java:141)
      at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:353)
      at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:135)
      at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.<init>(QuorumJournalManager.java:104)
      at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.<init>(QuorumJournalManager.java:93)
      ... 10 more

      I suggest that if quorum is up format should not fails.

      1. HDFS-4210.001.patch
        3 kB
        Charles Lamb
      2. HDFS-4210.002.patch
        4 kB
        John Zhuge
      3. HDFS-4210.003.patch
        4 kB
        John Zhuge
      4. HDFS-4210.004.patch
        4 kB
        John Zhuge

        Issue Links

          Activity

          Hide
          tlipcon Todd Lipcon added a comment -

          Currently we dont have any way to format a minority of nodes in the quorum. So, if we allowed you to format with missing nodes, then there'd be no easy way to format the new node when you add it (aside from rsync). Given that, we made the decision (for now) to only succeed formatting if all of the nodes are up.

          What's the use case where you want to start running with only 2 nodes up? It wouldn't provide any redundancy over just having 1 node.

          Show
          tlipcon Todd Lipcon added a comment - Currently we dont have any way to format a minority of nodes in the quorum. So, if we allowed you to format with missing nodes, then there'd be no easy way to format the new node when you add it (aside from rsync). Given that, we made the decision (for now) to only succeed formatting if all of the nodes are up. What's the use case where you want to start running with only 2 nodes up? It wouldn't provide any redundancy over just having 1 node.
          Hide
          tlipcon Todd Lipcon added a comment -

          Just noticed the NPE in your stack trace, though - that definitely seems worth fixing to give a proper DNS error.

          Show
          tlipcon Todd Lipcon added a comment - Just noticed the NPE in your stack trace, though - that definitely seems worth fixing to give a proper DNS error.
          Hide
          cmccabe Colin P. McCabe added a comment -

          It should definitely throw a more helpful exception than NullPointerException. However, I think the general idea that the quorum format should fail if some JournalNodes could not be formatted makes some sense. If some JournalNodes could not be formatted, the system is running at reduced redundancy. This could cause major problems down the road if we silently return success here.

          Can you write a script to wait until all nodes are accessible (keep pinging every second until you get through, or something like that)?

          Alternately, perhaps we could add a switch like -partial that would return success from a partial format as long as a quorum of JNs got formatted. But I don't think it should be the default...

          Show
          cmccabe Colin P. McCabe added a comment - It should definitely throw a more helpful exception than NullPointerException . However, I think the general idea that the quorum format should fail if some JournalNodes could not be formatted makes some sense. If some JournalNodes could not be formatted, the system is running at reduced redundancy. This could cause major problems down the road if we silently return success here. Can you write a script to wait until all nodes are accessible (keep pinging every second until you get through, or something like that)? Alternately, perhaps we could add a switch like -partial that would return success from a partial format as long as a quorum of JNs got formatted. But I don't think it should be the default...
          Hide
          dam_ned Damien Hardy added a comment -

          Actually this should not be the nominal case but when creating a cluster we often start by provisioning master nodes only (NN + ZK + JN + ZKFC + 1JTmr1), make them work together and then pay attention to worker nodes (DN + TTmr1).
          But ZK and JournaleNode need to be at least 3. So we choose to put them on one of our worker. And all config file are set in this way.

          This is just for convenient purpose.
          And yes a good idea would be a nice error message

          Show
          dam_ned Damien Hardy added a comment - Actually this should not be the nominal case but when creating a cluster we often start by provisioning master nodes only (NN + ZK + JN + ZKFC + 1JTmr1), make them work together and then pay attention to worker nodes (DN + TTmr1). But ZK and JournaleNode need to be at least 3. So we choose to put them on one of our worker. And all config file are set in this way. This is just for convenient purpose. And yes a good idea would be a nice error message
          Hide
          mbookman Matt Bookman added a comment -

          Would appreciate if the priority of this issue was revisited as this problem can surface in a much more important situation, namely during failover from Active NameNode to Standby NameNode.

          In some virtualized environments, when a virtual machine instance goes away, the DNS entry for it goes away immediately as well.

          Thus the following is observed on the Standby NameNode (hadoop-2.0.5-alpha) during failure of the Active NameNode:

          2013-07-03 21:28:40,576 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services required for active state
          2013-07-03 21:28:40,579 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Error encountered requiring NN shutdown. Shutting down immediately.
          java.lang.IllegalArgumentException: Unable to construct journal, qjournal://hadoop-mm:8485;hadoop-nn-0:8485;hadoop-nn-1:8485/hadoop
          at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1254)
          at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournals(FSEditLog.java:226)
          at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournalsForWrite(FSEditLog.java:193)
          at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startActiveServices(FSNamesystem.java:722)
          <etc>

          One can work around this problem by dropping an /etc/hosts file onto the Standby NameNode. Doing so allows the failover to proceed.

          But this creates a maintenance hassle, and it is not clear why a DNS failure should be treated any differently here than the general inability to connect to one of the quorum journal nodes.

          Show
          mbookman Matt Bookman added a comment - Would appreciate if the priority of this issue was revisited as this problem can surface in a much more important situation, namely during failover from Active NameNode to Standby NameNode. In some virtualized environments, when a virtual machine instance goes away, the DNS entry for it goes away immediately as well. Thus the following is observed on the Standby NameNode (hadoop-2.0.5-alpha) during failure of the Active NameNode: 2013-07-03 21:28:40,576 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services required for active state 2013-07-03 21:28:40,579 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Error encountered requiring NN shutdown. Shutting down immediately. java.lang.IllegalArgumentException: Unable to construct journal, qjournal://hadoop-mm:8485;hadoop-nn-0:8485;hadoop-nn-1:8485/hadoop at org.apache.hadoop.hdfs.server.namenode.FSEditLog.createJournal(FSEditLog.java:1254) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournals(FSEditLog.java:226) at org.apache.hadoop.hdfs.server.namenode.FSEditLog.initJournalsForWrite(FSEditLog.java:193) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startActiveServices(FSNamesystem.java:722) <etc> One can work around this problem by dropping an /etc/hosts file onto the Standby NameNode. Doing so allows the failover to proceed. But this creates a maintenance hassle, and it is not clear why a DNS failure should be treated any differently here than the general inability to connect to one of the quorum journal nodes.
          Hide
          cmccabe Colin P. McCabe added a comment -

          Hi Matt,

          I created a separate issue to discuss this in the context of failover rather than formatting. It seems easier to do that than to change the description on this one. This way the first few comments in this JIRA will make sense (they all talk about formatting, not failover.)

          Show
          cmccabe Colin P. McCabe added a comment - Hi Matt, I created a separate issue to discuss this in the context of failover rather than formatting. It seems easier to do that than to change the description on this one. This way the first few comments in this JIRA will make sense (they all talk about formatting, not failover.)
          Hide
          clamb Charles Lamb added a comment -

          The bottom line for this jira seems to be the produce a better exception than NPE when DNS resolution fails for any hosts named in the QJM. Any RuntimeException will cause the NN -format to fail in that case. The attached patch checks for a null pointer and throws an IllegalArgumentException instead. There's a unit test to go with it, although that doesn't test for NN format failure, but rather QJM startup failure.

          Show
          clamb Charles Lamb added a comment - The bottom line for this jira seems to be the produce a better exception than NPE when DNS resolution fails for any hosts named in the QJM. Any RuntimeException will cause the NN -format to fail in that case. The attached patch checks for a null pointer and throws an IllegalArgumentException instead. There's a unit test to go with it, although that doesn't test for NN format failure, but rather QJM startup failure.
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12682752/HDFS-4210.001.patch
          against trunk revision eb4045e.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8795//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8795//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12682752/HDFS-4210.001.patch against trunk revision eb4045e. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/8795//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8795//console This message is automatically generated.
          Hide
          clamb Charles Lamb added a comment -

          Trying to get Jenkins to listen to me.

          Show
          clamb Charles Lamb added a comment - Trying to get Jenkins to listen to me.
          Hide
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12682752/HDFS-4210.001.patch
          against trunk revision b375d1f.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs:

          org.apache.hadoop.tracing.TestTracing

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10027//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10027//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12682752/HDFS-4210.001.patch against trunk revision b375d1f. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.tracing.TestTracing Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/10027//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/10027//console This message is automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks all for the discussion so far, and Charles Lamb for the patch. I'm trying to resurrect this patch since it's almost done (I think).
          Nits:

          • Maybe we can throw a UnknownHostException instead of IllegalArgumentException?
          • I don't think we need to name this jira in testUnresolvableHostNameFailsGracefully's javadoc. One can always look it up in git history if interested.

          Other than that +1 (non-binding). Verified patch compile and test pass with latest trunk.

          Show
          xiaochen Xiao Chen added a comment - Thanks all for the discussion so far, and Charles Lamb for the patch. I'm trying to resurrect this patch since it's almost done (I think). Nits: Maybe we can throw a UnknownHostException instead of IllegalArgumentException? I don't think we need to name this jira in testUnresolvableHostNameFailsGracefully 's javadoc. One can always look it up in git history if interested. Other than that +1 (non-binding). Verified patch compile and test pass with latest trunk.
          Hide
          jzhuge John Zhuge added a comment -

          Charles Lamb/Charles Lamb, I am taking over the jira in order to push it over the finish line. Hope that is ok with you.

          Show
          jzhuge John Zhuge added a comment - Charles Lamb / Charles Lamb , I am taking over the jira in order to push it over the finish line. Hope that is ok with you.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 32s trunk passed
          +1 compile 1m 0s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 1m 12s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 2m 6s trunk passed
          +1 javadoc 0m 59s trunk passed
          +1 mvninstall 1m 3s the patch passed
          +1 compile 0m 44s the patch passed
          +1 javac 0m 44s the patch passed
          +1 checkstyle 0m 23s the patch passed
          +1 mvnsite 0m 51s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 51s the patch passed
          +1 javadoc 0m 56s the patch passed
          +1 unit 76m 41s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          99m 13s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12682752/HDFS-4210.001.patch
          JIRA Issue HDFS-4210
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6e1c2852c653 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a1f3293
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16533/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16533/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 32s trunk passed +1 compile 1m 0s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 12s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 2m 6s trunk passed +1 javadoc 0m 59s trunk passed +1 mvninstall 1m 3s the patch passed +1 compile 0m 44s the patch passed +1 javac 0m 44s the patch passed +1 checkstyle 0m 23s the patch passed +1 mvnsite 0m 51s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 51s the patch passed +1 javadoc 0m 56s the patch passed +1 unit 76m 41s hadoop-hdfs in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 99m 13s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12682752/HDFS-4210.001.patch JIRA Issue HDFS-4210 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6e1c2852c653 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a1f3293 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16533/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16533/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          clamb Charles Lamb added a comment -

          I am taking over the jira in order to push it over the finish line. Hope that is ok with you.

          No problem John Zhuge. Thanks for taking over.

          Show
          clamb Charles Lamb added a comment - I am taking over the jira in order to push it over the finish line. Hope that is ok with you. No problem John Zhuge . Thanks for taking over.
          Hide
          jzhuge John Zhuge added a comment - - edited

          Discovered that exception UnknownHostException already thrown and caught in the following call path earlier than the NPE call path listed in JIRA Description:

          	  at org.apache.hadoop.net.NetUtils.createSocketAddrForHost(NetUtils.java:245)
          	  at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:217)
          	  at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164)
          	  at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.getLoggerAddresses(QuorumJournalManager.java:390)
          	  at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:364)
          	  at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:149)
          	  at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.<init>(QuorumJournalManager.java:116)
          	  at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.<init>(QuorumJournalManager.java:105)
          	  at org.apache.hadoop.hdfs.qjournal.client.TestQJMWithFaults.testUnresolvableHostNameFailsGracefully(TestQJMWithFaults.java:201)
          
          'createSocketAddrForHost'
              } catch (UnknownHostException e) {
                addr = InetSocketAddress.createUnresolved(host, port);
              }
          

          createSocketAddrForHost swallows the UHE exception and creates an unsolved InetSocketAddress. Callers are supposed to check with isUnresolved.

          getLoggerAddresses is the earliest opportunity to throw UHE.

          Show
          jzhuge John Zhuge added a comment - - edited Discovered that exception UnknownHostException already thrown and caught in the following call path earlier than the NPE call path listed in JIRA Description: at org.apache.hadoop.net.NetUtils.createSocketAddrForHost(NetUtils.java:245) at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:217) at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:164) at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.getLoggerAddresses(QuorumJournalManager.java:390) at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:364) at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.createLoggers(QuorumJournalManager.java:149) at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.<init>(QuorumJournalManager.java:116) at org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.<init>(QuorumJournalManager.java:105) at org.apache.hadoop.hdfs.qjournal.client.TestQJMWithFaults.testUnresolvableHostNameFailsGracefully(TestQJMWithFaults.java:201) 'createSocketAddrForHost' } catch (UnknownHostException e) { addr = InetSocketAddress.createUnresolved(host, port); } createSocketAddrForHost swallows the UHE exception and creates an unsolved InetSocketAddress . Callers are supposed to check with isUnresolved . getLoggerAddresses is the earliest opportunity to throw UHE.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 002:

          • Throw UnknownHostException earlier in getLoggerAddresses when a JN hostname can not be resolved
          • Use JUnit rule ExpectedException
          Show
          jzhuge John Zhuge added a comment - Patch 002: Throw UnknownHostException earlier in getLoggerAddresses when a JN hostname can not be resolved Use JUnit rule ExpectedException
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 1s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 46s trunk passed
          +1 compile 0m 44s trunk passed
          +1 checkstyle 0m 25s trunk passed
          +1 mvnsite 0m 51s trunk passed
          +1 mvneclipse 0m 12s trunk passed
          +1 findbugs 1m 41s trunk passed
          +1 javadoc 0m 54s trunk passed
          +1 mvninstall 0m 44s the patch passed
          +1 compile 0m 42s the patch passed
          +1 javac 0m 42s the patch passed
          -0 checkstyle 0m 22s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 32 unchanged - 0 fixed = 34 total (was 32)
          +1 mvnsite 0m 48s the patch passed
          +1 mvneclipse 0m 9s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 46s the patch passed
          +1 javadoc 0m 52s the patch passed
          -1 unit 61m 21s hadoop-hdfs in the patch failed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          80m 11s



          Reason Tests
          Failed junit tests hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-4210
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825950/HDFS-4210.002.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux ab309d46ce92 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c258171
          Default Java 1.8.0_101
          findbugs v3.0.0
          checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16557/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
          unit https://builds.apache.org/job/PreCommit-HDFS-Build/16557/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16557/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16557/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 1s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 46s trunk passed +1 compile 0m 44s trunk passed +1 checkstyle 0m 25s trunk passed +1 mvnsite 0m 51s trunk passed +1 mvneclipse 0m 12s trunk passed +1 findbugs 1m 41s trunk passed +1 javadoc 0m 54s trunk passed +1 mvninstall 0m 44s the patch passed +1 compile 0m 42s the patch passed +1 javac 0m 42s the patch passed -0 checkstyle 0m 22s hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 32 unchanged - 0 fixed = 34 total (was 32) +1 mvnsite 0m 48s the patch passed +1 mvneclipse 0m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 46s the patch passed +1 javadoc 0m 52s the patch passed -1 unit 61m 21s hadoop-hdfs in the patch failed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 80m 11s Reason Tests Failed junit tests hadoop.hdfs.server.blockmanagement.TestRBWBlockInvalidation Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-4210 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825950/HDFS-4210.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux ab309d46ce92 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c258171 Default Java 1.8.0_101 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-HDFS-Build/16557/artifact/patchprocess/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt unit https://builds.apache.org/job/PreCommit-HDFS-Build/16557/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16557/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16557/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 003:

          • Fix checkstyle

          TestRBWBlockInvalidation failure for 002 passes locally.

          Show
          jzhuge John Zhuge added a comment - Patch 003: Fix checkstyle TestRBWBlockInvalidation failure for 002 passes locally.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
          +1 @author 0m 1s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 53s trunk passed
          +1 compile 0m 52s trunk passed
          +1 checkstyle 0m 34s trunk passed
          +1 mvnsite 1m 8s trunk passed
          +1 mvneclipse 0m 15s trunk passed
          +1 findbugs 2m 4s trunk passed
          +1 javadoc 1m 5s trunk passed
          +1 mvninstall 1m 1s the patch passed
          +1 compile 0m 57s the patch passed
          +1 javac 0m 57s the patch passed
          +1 checkstyle 0m 28s the patch passed
          +1 mvnsite 1m 3s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 3s the patch passed
          +1 javadoc 1m 4s the patch passed
          +1 unit 76m 45s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 21s The patch does not generate ASF License warnings.
          100m 43s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-4210
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826017/HDFS-4210.003.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 79237b8fd089 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c258171
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16566/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16566/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated. +1 @author 0m 1s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 53s trunk passed +1 compile 0m 52s trunk passed +1 checkstyle 0m 34s trunk passed +1 mvnsite 1m 8s trunk passed +1 mvneclipse 0m 15s trunk passed +1 findbugs 2m 4s trunk passed +1 javadoc 1m 5s trunk passed +1 mvninstall 1m 1s the patch passed +1 compile 0m 57s the patch passed +1 javac 0m 57s the patch passed +1 checkstyle 0m 28s the patch passed +1 mvnsite 1m 3s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 3s the patch passed +1 javadoc 1m 4s the patch passed +1 unit 76m 45s hadoop-hdfs in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 100m 43s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-4210 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826017/HDFS-4210.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 79237b8fd089 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c258171 Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16566/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16566/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          jzhuge John Zhuge added a comment -

          Change the title to reflect the intended fix and be consistent with HDFS-4957.

          Show
          jzhuge John Zhuge added a comment - Change the title to reflect the intended fix and be consistent with HDFS-4957 .
          Hide
          xiaochen Xiao Chen added a comment -

          Thanks John for the patches.

          2 comments, other than these LGTM:

                if (isa.isUnresolved()) {
                  throw new UnknownHostException(isa.getHostName());
                }
          

          Prefer what Charles Lamb did: toString instead of getHostName. The latter may end up resolve an IP address again.

            private static URI createUnresolvableURI(String jid) throws URISyntaxException {
              try {
                return new URI("qjournal://" + "bogus:12345" + "/" + jid);
              } catch (URISyntaxException e) {
                throw new AssertionError(e);
              }
            }
          

          Since it's a test, it maybe clearer if we just let it throw here.

          Show
          xiaochen Xiao Chen added a comment - Thanks John for the patches. 2 comments, other than these LGTM: if (isa.isUnresolved()) { throw new UnknownHostException(isa.getHostName()); } Prefer what Charles Lamb did: toString instead of getHostName . The latter may end up resolve an IP address again. private static URI createUnresolvableURI( String jid) throws URISyntaxException { try { return new URI( "qjournal: //" + "bogus:12345" + "/" + jid); } catch (URISyntaxException e) { throw new AssertionError(e); } } Since it's a test, it maybe clearer if we just let it throw here.
          Hide
          jzhuge John Zhuge added a comment -

          Patch 004:

          • Xiao's comments
          Show
          jzhuge John Zhuge added a comment - Patch 004: Xiao's comments
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 16s trunk passed
          +1 compile 0m 52s trunk passed
          +1 checkstyle 0m 28s trunk passed
          +1 mvnsite 1m 0s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 1m 50s trunk passed
          +1 javadoc 0m 56s trunk passed
          +1 mvninstall 0m 52s the patch passed
          +1 compile 0m 51s the patch passed
          +1 javac 0m 51s the patch passed
          +1 checkstyle 0m 26s the patch passed
          +1 mvnsite 0m 57s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 2m 0s the patch passed
          +1 javadoc 0m 58s the patch passed
          +1 unit 65m 46s hadoop-hdfs in the patch passed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          87m 34s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue HDFS-4210
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826078/HDFS-4210.004.patch
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 0ba0e99f56cd 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 5d1609d
          Default Java 1.8.0_101
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16568/testReport/
          modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
          Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16568/console
          Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 16s trunk passed +1 compile 0m 52s trunk passed +1 checkstyle 0m 28s trunk passed +1 mvnsite 1m 0s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 1m 50s trunk passed +1 javadoc 0m 56s trunk passed +1 mvninstall 0m 52s the patch passed +1 compile 0m 51s the patch passed +1 javac 0m 51s the patch passed +1 checkstyle 0m 26s the patch passed +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 2m 0s the patch passed +1 javadoc 0m 58s the patch passed +1 unit 65m 46s hadoop-hdfs in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 87m 34s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue HDFS-4210 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12826078/HDFS-4210.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0ba0e99f56cd 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28 20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 5d1609d Default Java 1.8.0_101 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-HDFS-Build/16568/testReport/ modules C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs Console output https://builds.apache.org/job/PreCommit-HDFS-Build/16568/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          xiaochen Xiao Chen added a comment -

          +1 on patch 4, committing soon.

          Show
          xiaochen Xiao Chen added a comment - +1 on patch 4, committing soon.
          Hide
          xiaochen Xiao Chen added a comment -

          I have committed this to trunk, branch-2 and branch-2.8. Thanks Damien Hardy for creating the issue, Charles Lamb + John Zhuge for the patch, and all for the comments!

          Show
          xiaochen Xiao Chen added a comment - I have committed this to trunk, branch-2 and branch-2.8. Thanks Damien Hardy for creating the issue, Charles Lamb + John Zhuge for the patch, and all for the comments!
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10372 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10372/)
          HDFS-4210. Throw helpful exception when DNS entry for JournalNode cannot (xiao: rev cd5e10ccca9b919d55ef7a500e61b44bd1a00171)

          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java
          • (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQJMWithFaults.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10372 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10372/ ) HDFS-4210 . Throw helpful exception when DNS entry for JournalNode cannot (xiao: rev cd5e10ccca9b919d55ef7a500e61b44bd1a00171) (edit) hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/QuorumJournalManager.java (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/client/TestQJMWithFaults.java
          Hide
          jzhuge John Zhuge added a comment -

          Thanks @xiao for the review and commit!

          Show
          jzhuge John Zhuge added a comment - Thanks @xiao for the review and commit!

            People

            • Assignee:
              jzhuge John Zhuge
              Reporter:
              dam_ned Damien Hardy
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development