Hadoop Common
  1. Hadoop Common
  2. HADOOP-9369

DNS#reverseDns() can return hostname with . appended at the end

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.1, 2.0.3-alpha, 0.23.5
    • Fix Version/s: 1.2.0, 2.1.0-beta
    • Component/s: net
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      DNS#reverseDns uses javax.naming.InitialDirContext to do a reverse DNS lookup. This can sometimes return hostnames with a . at the end.

      Saw this happen on hadoop-1: two nodes with tasktracker.dns.interface set to eth0

      1. hadoop-9369.patch
        0.8 kB
        Karthik Kambatla
      2. hadoop-9369-branch1.patch
        0.6 kB
        Karthik Kambatla
      3. hadoop-9369.patch
        0.8 kB
        Karthik Kambatla

        Issue Links

          Activity

          Karthik Kambatla (Inactive) created issue -
          Hide
          Karthik Kambatla (Inactive) added a comment -

          The fix is fairly straight-forward as documented here http://stackoverflow.com/questions/7097623/need-to-perform-a-reverse-dns-lookup-of-a-particular-ip-address-in-java

          • Check if the hostname has a '.' at the end
          Show
          Karthik Kambatla (Inactive) added a comment - The fix is fairly straight-forward as documented here http://stackoverflow.com/questions/7097623/need-to-perform-a-reverse-dns-lookup-of-a-particular-ip-address-in-java Check if the hostname has a '.' at the end
          Karthik Kambatla (Inactive) made changes -
          Field Original Value New Value
          Attachment hadoop-9369.patch [ 12572260 ]
          Karthik Kambatla (Inactive) made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Karthik Kambatla (Inactive) made changes -
          Attachment hadoop-9369-branch1.patch [ 12572262 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12572262/hadoop-9369-branch1.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2273//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572262/hadoop-9369-branch1.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2273//console This message is automatically generated.
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Ran a 2-node cluster, one with the patch and one without. Logged the hostname: the one with the patch doesn't have a '.' while the other one has.

          If one were to write a unit test for this, we should either (1) mock InitialDirContext's getAttributes method (it is not static either) or (2) write a singleton wrapper for InitialDirContext and modify the method in the context of the test. Manual verification validates the fix.

          Show
          Karthik Kambatla (Inactive) added a comment - Ran a 2-node cluster, one with the patch and one without. Logged the hostname: the one with the patch doesn't have a '.' while the other one has. If one were to write a unit test for this, we should either (1) mock InitialDirContext's getAttributes method (it is not static either) or (2) write a singleton wrapper for InitialDirContext and modify the method in the context of the test. Manual verification validates the fix.
          Karthik Kambatla (Inactive) made changes -
          Attachment hadoop-9369.patch [ 12572263 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12572263/hadoop-9369.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2274//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2274//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572263/hadoop-9369.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2274//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2274//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          Hi Karthik, the patch seems fine to me, and I agree that the patch is so simple and writing a test sufficiently difficult that it seems unnecessary to write a test for this.

          One question - can you comment on the ramifications of this issue? How does it manifest itself? And what triggers it?

          Show
          Aaron T. Myers added a comment - Hi Karthik, the patch seems fine to me, and I agree that the patch is so simple and writing a test sufficiently difficult that it seems unnecessary to write a test for this. One question - can you comment on the ramifications of this issue? How does it manifest itself? And what triggers it?
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Thanks Aaron for taking a look at the patch.

          Consider a cluster setup where hadoop_allowed_hosts.txt is used to list nodes that can be included, say nodes are 'sample1', 'sample2'. Now, if they also choose to fix the interface to be used (say eth0), the TTs during initialization call themselves 'sample1.' and 'sample2.', note the '.' at the end of the hostnames. Now, as this doesn't match any of the nodes listed in the file, the two TTs are disallowed to join. Ideally, the TTs should identify themselves as correct hostnames without '.' at the end.

          The code path that the patch modifies is invoked only when we specify a particular interface, otherwise the default/canonical hostnames are picked. As for when InitialDirContext#getAttributes() returns a hostname with '.' at the end, I am not quite certain - it didn't show up on a pseudo-distributed test.

          Show
          Karthik Kambatla (Inactive) added a comment - Thanks Aaron for taking a look at the patch. Consider a cluster setup where hadoop_allowed_hosts.txt is used to list nodes that can be included, say nodes are 'sample1', 'sample2'. Now, if they also choose to fix the interface to be used (say eth0), the TTs during initialization call themselves 'sample1.' and 'sample2.', note the '.' at the end of the hostnames. Now, as this doesn't match any of the nodes listed in the file, the two TTs are disallowed to join. Ideally, the TTs should identify themselves as correct hostnames without '.' at the end. The code path that the patch modifies is invoked only when we specify a particular interface, otherwise the default/canonical hostnames are picked. As for when InitialDirContext#getAttributes() returns a hostname with '.' at the end, I am not quite certain - it didn't show up on a pseudo-distributed test.
          Hide
          Aaron T. Myers added a comment -

          Thanks a lot for the explanation, Karthik. Makes sense, and the patch seems quite straightforward.

          +1, I'm going to commit this momentarily.

          Show
          Aaron T. Myers added a comment - Thanks a lot for the explanation, Karthik. Makes sense, and the patch seems quite straightforward. +1, I'm going to commit this momentarily.
          Hide
          Aaron T. Myers added a comment -

          I've just committed this to trunk, branch-2, and branch-1.

          Thanks a lot for the contribution, Karthik.

          Show
          Aaron T. Myers added a comment - I've just committed this to trunk, branch-2, and branch-1. Thanks a lot for the contribution, Karthik.
          Aaron T. Myers made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 1.2.0 [ 12321659 ]
          Fix Version/s 2.0.5-beta [ 12324030 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3438 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3438/)
          Move CHANGES.txt entry for HADOOP-9369 to the correct place. (Revision 1454174)
          HADOOP-9369. DNS#reverseDns() can return hostname with . appended at the end. Contributed by Karthik Kambatla. (Revision 1454172)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454174
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454172
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNS.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3438 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3438/ ) Move CHANGES.txt entry for HADOOP-9369 to the correct place. (Revision 1454174) HADOOP-9369 . DNS#reverseDns() can return hostname with . appended at the end. Contributed by Karthik Kambatla. (Revision 1454172) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454174 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454172 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNS.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #149 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/149/)
          Move CHANGES.txt entry for HADOOP-9369 to the correct place. (Revision 1454174)
          HADOOP-9369. DNS#reverseDns() can return hostname with . appended at the end. Contributed by Karthik Kambatla. (Revision 1454172)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454174
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454172
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNS.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #149 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/149/ ) Move CHANGES.txt entry for HADOOP-9369 to the correct place. (Revision 1454174) HADOOP-9369 . DNS#reverseDns() can return hostname with . appended at the end. Contributed by Karthik Kambatla. (Revision 1454172) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454174 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454172 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNS.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1338 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1338/)
          Move CHANGES.txt entry for HADOOP-9369 to the correct place. (Revision 1454174)
          HADOOP-9369. DNS#reverseDns() can return hostname with . appended at the end. Contributed by Karthik Kambatla. (Revision 1454172)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454174
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454172
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNS.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1338 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1338/ ) Move CHANGES.txt entry for HADOOP-9369 to the correct place. (Revision 1454174) HADOOP-9369 . DNS#reverseDns() can return hostname with . appended at the end. Contributed by Karthik Kambatla. (Revision 1454172) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454174 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454172 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNS.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1366 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1366/)
          Move CHANGES.txt entry for HADOOP-9369 to the correct place. (Revision 1454174)
          HADOOP-9369. DNS#reverseDns() can return hostname with . appended at the end. Contributed by Karthik Kambatla. (Revision 1454172)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454174
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454172
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNS.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1366 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1366/ ) Move CHANGES.txt entry for HADOOP-9369 to the correct place. (Revision 1454174) HADOOP-9369 . DNS#reverseDns() can return hostname with . appended at the end. Contributed by Karthik Kambatla. (Revision 1454172) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454174 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454172 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNS.java
          Eli Collins made changes -
          Link This issue relates to HBASE-7693 [ HBASE-7693 ]
          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop 1.2.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.
          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Gavin made changes -
          Assignee Karthik Kambatla [ kkambatl ] Karthik Kambatla [ kasha ]
          Gavin made changes -
          Reporter Karthik Kambatla [ kkambatl ] Karthik Kambatla [ kasha ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          2h 6m 1 Karthik Kambatla (Inactive) 06/Mar/13 05:25
          Patch Available Patch Available Resolved Resolved
          1d 18h 31m 1 Aaron T. Myers 07/Mar/13 23:57
          Resolved Resolved Closed Closed
          68d 5h 18m 1 Matt Foley 15/May/13 05:15

            People

            • Assignee:
              Karthik Kambatla
              Reporter:
              Karthik Kambatla
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development