Hadoop Common
  1. Hadoop Common
  2. HADOOP-6682

NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: io
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      public static String normalizeHostName(String name) {
      if (Character.digit(name.charAt(0), 16) != -1) {
      return name;

      This code is attempting to short-circuit the hostname->ip resolution on the assumption that if name starts with a digit, it's already an ip address. This is of questionable value, but because it checks for a hex digit, it will fail on names starting with [a-f]. Such names will not be converted to an ip address, but be returned unchanged.

      1. HADOOP-6682.patch
        0.6 kB
        Jakob Homan
      2. HADOOP-6682-Y20.patch
        0.6 kB
        Jakob Homan

        Issue Links

          Activity

          Hide
          Hong Tang added a comment -

          Similar issue came up in MAPREDUCE-1222. The following code might be useful:

          +  static Pattern IP_PATTERN;
          +  
          +  static {
          +    // 0-255
          +    String IPV4BK1 = "(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)";
          +    // .b.c.d - where b/c/d are 0-255, and optionally adding two more
          +    // backslashes before each period
          +    String IPV4BKN = "(?:\\\\?\\." + IPV4BK1 + "){3}";
          +    String IPV4_PATTERN = IPV4BK1 + IPV4BKN;
          +    
          +    // first hexadecimal number
          +    String IPV6BK1 = "(?:[0-9a-fA-F]{1,4})";
          +    // remaining 7 hexadecimal numbers, each preceded with ":".
          +    String IPV6BKN = "(?::" + IPV6BK1 + "){7}";
          +    String IPV6_PATTERN = IPV6BK1 + IPV6BKN;
          +
          +    IP_PATTERN = Pattern.compile(
          +        "^(?:" + IPV4_PATTERN + "|" + IPV6_PATTERN + ")$");
          +  }
          +
          + 
          +  static boolean isIPAddress(String hostname) {
          +    return IP_PATTERN.matcher(hostname).matches();
          +  }
          
          Show
          Hong Tang added a comment - Similar issue came up in MAPREDUCE-1222 . The following code might be useful: + static Pattern IP_PATTERN; + + static { + // 0-255 + String IPV4BK1 = "(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)"; + // .b.c.d - where b/c/d are 0-255, and optionally adding two more + // backslashes before each period + String IPV4BKN = "(?:\\\\?\\." + IPV4BK1 + "){3}"; + String IPV4_PATTERN = IPV4BK1 + IPV4BKN; + + // first hexadecimal number + String IPV6BK1 = "(?:[0-9a-fA-F]{1,4})"; + // remaining 7 hexadecimal numbers, each preceded with ":". + String IPV6BKN = "(?::" + IPV6BK1 + "){7}"; + String IPV6_PATTERN = IPV6BK1 + IPV6BKN; + + IP_PATTERN = Pattern.compile( + "^(?:" + IPV4_PATTERN + "|" + IPV6_PATTERN + ")$"); + } + + + static boolean isIPAddress(String hostname) { + return IP_PATTERN.matcher(hostname).matches(); + }
          Hide
          Hong Tang added a comment -

          The regex IPV4BKN needs some change because in job history dots (.) are escaped with double backslashes.

          Show
          Hong Tang added a comment - The regex IPV4BKN needs some change because in job history dots (.) are escaped with double backslashes.
          Hide
          Jakob Homan added a comment -

          Patch for Y20, which we forgot to upload at the time. Not for commit.

          Show
          Jakob Homan added a comment - Patch for Y20, which we forgot to upload at the time. Not for commit.
          Hide
          Jakob Homan added a comment -

          Patch for trunk. Same as Y20 patch. Just changes the base to 10 rather than 16.

          Hong, I like your approach and will open another JIRA to address the general shortcoming of the normalizeHostname function. For this one I'd like to take as little risk as possible.

          I considered writing a unit test for the fix that would try to normalize some host name that starts with an offending hostname (ask.com), but this would violate the unit-test-shouldn't-talk-to-outside-resources guideline, and couldn't think of a better way to test it. If someone has one, let me and I'll add it. Otherwise, I think it's good to go without one.

          Show
          Jakob Homan added a comment - Patch for trunk. Same as Y20 patch. Just changes the base to 10 rather than 16. Hong, I like your approach and will open another JIRA to address the general shortcoming of the normalizeHostname function. For this one I'd like to take as little risk as possible. I considered writing a unit test for the fix that would try to normalize some host name that starts with an offending hostname (ask.com), but this would violate the unit-test-shouldn't-talk-to-outside-resources guideline, and couldn't think of a better way to test it. If someone has one, let me and I'll add it. Otherwise, I think it's good to go without one.
          Hide
          Jakob Homan added a comment -

          submitting patch.

          Show
          Jakob Homan added a comment - submitting patch.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12446933/HADOOP-6682.patch
          against trunk revision 953910.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/79/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/79/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/79/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/79/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446933/HADOOP-6682.patch against trunk revision 953910. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/79/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/79/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/79/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/79/console This message is automatically generated.
          Hide
          Devaraj Das added a comment -

          +1

          Show
          Devaraj Das added a comment - +1
          Hide
          Jakob Homan added a comment -

          I call bogus on the javadoc warning, since this patch doesn't affect that. Re-run Hudson for test failure. I don't see it on my machine, but let's see if it re-occurs.

          Show
          Jakob Homan added a comment - I call bogus on the javadoc warning, since this patch doesn't affect that. Re-run Hudson for test failure. I don't see it on my machine, but let's see if it re-occurs.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12446933/HADOOP-6682.patch
          against trunk revision 953910.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          -1 javadoc. The javadoc tool appears to have generated 1 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/81/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/81/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/81/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/81/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12446933/HADOOP-6682.patch against trunk revision 953910. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/81/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/81/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/81/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h1.grid.sp2.yahoo.net/81/console This message is automatically generated.
          Hide
          Jakob Homan added a comment -

          I've committed this. Resolving as fixed.

          Show
          Jakob Homan added a comment - I've committed this. Resolving as fixed.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #301 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/301/)
          HADOOP-6682. NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #301 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk-Commit/301/ ) HADOOP-6682 . NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #364 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/364/)
          HADOOP-6682. NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #364 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Common-trunk/364/ ) HADOOP-6682 . NetUtils:normalizeHostName does not process hostnames starting with [a-f] correctly.
          Hide
          Eli Collins added a comment -

          The Y20 patch here is bogus btw it changes from 10 to 16 instead of vice versa. Doesn't look like it was applied anywhere so not a big deal.

          Show
          Eli Collins added a comment - The Y20 patch here is bogus btw it changes from 10 to 16 instead of vice versa. Doesn't look like it was applied anywhere so not a big deal.
          Hide
          Jakob Homan added a comment -

          hmmm... must have run the patch the wrong way. That was one crazy week.

          Show
          Jakob Homan added a comment - hmmm... must have run the patch the wrong way. That was one crazy week.

            People

            • Assignee:
              Jakob Homan
              Reporter:
              Jakob Homan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development