Hadoop Common
  1. Hadoop Common
  2. HADOOP-9369

DNS#reverseDns() can return hostname with . appended at the end

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.1, 2.0.3-alpha, 0.23.5
    • Fix Version/s: 1.2.0, 2.1.0-beta
    • Component/s: net
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      DNS#reverseDns uses javax.naming.InitialDirContext to do a reverse DNS lookup. This can sometimes return hostnames with a . at the end.

      Saw this happen on hadoop-1: two nodes with tasktracker.dns.interface set to eth0

      1. hadoop-9369.patch
        0.8 kB
        Karthik Kambatla
      2. hadoop-9369-branch1.patch
        0.6 kB
        Karthik Kambatla
      3. hadoop-9369.patch
        0.8 kB
        Karthik Kambatla

        Issue Links

          Activity

          Hide
          Matt Foley added a comment -

          Closed upon release of Hadoop 1.2.0.

          Show
          Matt Foley added a comment - Closed upon release of Hadoop 1.2.0.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1366 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1366/)
          Move CHANGES.txt entry for HADOOP-9369 to the correct place. (Revision 1454174)
          HADOOP-9369. DNS#reverseDns() can return hostname with . appended at the end. Contributed by Karthik Kambatla. (Revision 1454172)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454174
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454172
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNS.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1366 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1366/ ) Move CHANGES.txt entry for HADOOP-9369 to the correct place. (Revision 1454174) HADOOP-9369 . DNS#reverseDns() can return hostname with . appended at the end. Contributed by Karthik Kambatla. (Revision 1454172) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454174 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454172 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNS.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1338 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1338/)
          Move CHANGES.txt entry for HADOOP-9369 to the correct place. (Revision 1454174)
          HADOOP-9369. DNS#reverseDns() can return hostname with . appended at the end. Contributed by Karthik Kambatla. (Revision 1454172)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454174
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454172
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNS.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1338 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1338/ ) Move CHANGES.txt entry for HADOOP-9369 to the correct place. (Revision 1454174) HADOOP-9369 . DNS#reverseDns() can return hostname with . appended at the end. Contributed by Karthik Kambatla. (Revision 1454172) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454174 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454172 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNS.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Yarn-trunk #149 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/149/)
          Move CHANGES.txt entry for HADOOP-9369 to the correct place. (Revision 1454174)
          HADOOP-9369. DNS#reverseDns() can return hostname with . appended at the end. Contributed by Karthik Kambatla. (Revision 1454172)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454174
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454172
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNS.java
          Show
          Hudson added a comment - Integrated in Hadoop-Yarn-trunk #149 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/149/ ) Move CHANGES.txt entry for HADOOP-9369 to the correct place. (Revision 1454174) HADOOP-9369 . DNS#reverseDns() can return hostname with . appended at the end. Contributed by Karthik Kambatla. (Revision 1454172) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454174 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454172 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNS.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-trunk-Commit #3438 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3438/)
          Move CHANGES.txt entry for HADOOP-9369 to the correct place. (Revision 1454174)
          HADOOP-9369. DNS#reverseDns() can return hostname with . appended at the end. Contributed by Karthik Kambatla. (Revision 1454172)

          Result = SUCCESS
          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454174
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454172
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNS.java
          Show
          Hudson added a comment - Integrated in Hadoop-trunk-Commit #3438 (See https://builds.apache.org/job/Hadoop-trunk-Commit/3438/ ) Move CHANGES.txt entry for HADOOP-9369 to the correct place. (Revision 1454174) HADOOP-9369 . DNS#reverseDns() can return hostname with . appended at the end. Contributed by Karthik Kambatla. (Revision 1454172) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454174 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1454172 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/DNS.java
          Hide
          Aaron T. Myers added a comment -

          I've just committed this to trunk, branch-2, and branch-1.

          Thanks a lot for the contribution, Karthik.

          Show
          Aaron T. Myers added a comment - I've just committed this to trunk, branch-2, and branch-1. Thanks a lot for the contribution, Karthik.
          Hide
          Aaron T. Myers added a comment -

          Thanks a lot for the explanation, Karthik. Makes sense, and the patch seems quite straightforward.

          +1, I'm going to commit this momentarily.

          Show
          Aaron T. Myers added a comment - Thanks a lot for the explanation, Karthik. Makes sense, and the patch seems quite straightforward. +1, I'm going to commit this momentarily.
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Thanks Aaron for taking a look at the patch.

          Consider a cluster setup where hadoop_allowed_hosts.txt is used to list nodes that can be included, say nodes are 'sample1', 'sample2'. Now, if they also choose to fix the interface to be used (say eth0), the TTs during initialization call themselves 'sample1.' and 'sample2.', note the '.' at the end of the hostnames. Now, as this doesn't match any of the nodes listed in the file, the two TTs are disallowed to join. Ideally, the TTs should identify themselves as correct hostnames without '.' at the end.

          The code path that the patch modifies is invoked only when we specify a particular interface, otherwise the default/canonical hostnames are picked. As for when InitialDirContext#getAttributes() returns a hostname with '.' at the end, I am not quite certain - it didn't show up on a pseudo-distributed test.

          Show
          Karthik Kambatla (Inactive) added a comment - Thanks Aaron for taking a look at the patch. Consider a cluster setup where hadoop_allowed_hosts.txt is used to list nodes that can be included, say nodes are 'sample1', 'sample2'. Now, if they also choose to fix the interface to be used (say eth0), the TTs during initialization call themselves 'sample1.' and 'sample2.', note the '.' at the end of the hostnames. Now, as this doesn't match any of the nodes listed in the file, the two TTs are disallowed to join. Ideally, the TTs should identify themselves as correct hostnames without '.' at the end. The code path that the patch modifies is invoked only when we specify a particular interface, otherwise the default/canonical hostnames are picked. As for when InitialDirContext#getAttributes() returns a hostname with '.' at the end, I am not quite certain - it didn't show up on a pseudo-distributed test.
          Hide
          Aaron T. Myers added a comment -

          Hi Karthik, the patch seems fine to me, and I agree that the patch is so simple and writing a test sufficiently difficult that it seems unnecessary to write a test for this.

          One question - can you comment on the ramifications of this issue? How does it manifest itself? And what triggers it?

          Show
          Aaron T. Myers added a comment - Hi Karthik, the patch seems fine to me, and I agree that the patch is so simple and writing a test sufficiently difficult that it seems unnecessary to write a test for this. One question - can you comment on the ramifications of this issue? How does it manifest itself? And what triggers it?
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12572263/hadoop-9369.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2274//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2274//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572263/hadoop-9369.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/2274//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2274//console This message is automatically generated.
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Ran a 2-node cluster, one with the patch and one without. Logged the hostname: the one with the patch doesn't have a '.' while the other one has.

          If one were to write a unit test for this, we should either (1) mock InitialDirContext's getAttributes method (it is not static either) or (2) write a singleton wrapper for InitialDirContext and modify the method in the context of the test. Manual verification validates the fix.

          Show
          Karthik Kambatla (Inactive) added a comment - Ran a 2-node cluster, one with the patch and one without. Logged the hostname: the one with the patch doesn't have a '.' while the other one has. If one were to write a unit test for this, we should either (1) mock InitialDirContext's getAttributes method (it is not static either) or (2) write a singleton wrapper for InitialDirContext and modify the method in the context of the test. Manual verification validates the fix.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12572262/hadoop-9369-branch1.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2273//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12572262/hadoop-9369-branch1.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/2273//console This message is automatically generated.
          Hide
          Karthik Kambatla (Inactive) added a comment -

          The fix is fairly straight-forward as documented here http://stackoverflow.com/questions/7097623/need-to-perform-a-reverse-dns-lookup-of-a-particular-ip-address-in-java

          • Check if the hostname has a '.' at the end
          Show
          Karthik Kambatla (Inactive) added a comment - The fix is fairly straight-forward as documented here http://stackoverflow.com/questions/7097623/need-to-perform-a-reverse-dns-lookup-of-a-particular-ip-address-in-java Check if the hostname has a '.' at the end

            People

            • Assignee:
              Karthik Kambatla
              Reporter:
              Karthik Kambatla
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development