Hadoop Common
  1. Hadoop Common
  2. HADOOP-7499

Add method for doing a sanity check on hostnames in NetUtils

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.23.0
    • Fix Version/s: 0.23.0
    • Component/s: util
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      As part of MAPREDUCE-2489, we need a method in NetUtils to do a sanity check on hostnames

      1. patchJavadocWarnings.txt
        18 kB
        Jeffrey Naisbitt
      2. HADOOP-7499-v3.patch
        10 kB
        Jeffrey Naisbitt
      3. HADOOP-7499-v2.patch
        4 kB
        Jeffrey Naisbitt
      4. HADOOP-7499.patch
        3 kB
        Jeffrey Naisbitt

        Issue Links

          Activity

          Hide
          Jeffrey Naisbitt added a comment -

          This blocks the patch for trunk on MAPREDUCE-2489

          Show
          Jeffrey Naisbitt added a comment - This blocks the patch for trunk on MAPREDUCE-2489
          Hide
          Jeffrey Naisbitt added a comment -

          Manual test-patch.sh results

          -1 overall.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 javadoc. The javadoc tool appears to have generated 18 warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 system test framework. The patch passed system test framework compile.

          Show
          Jeffrey Naisbitt added a comment - Manual test-patch.sh results -1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 18 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 system test framework. The patch passed system test framework compile.
          Hide
          Jeffrey Naisbitt added a comment -

          I didn't see any new problems in the javadoc output, so I'm not sure where the -1 is coming from there. I didn't see anything related to this patch.

          Show
          Jeffrey Naisbitt added a comment - I didn't see any new problems in the javadoc output, so I'm not sure where the -1 is coming from there. I didn't see anything related to this patch.
          Hide
          Allen Wittenauer added a comment -

          Does the test actually do a DNS lookup?

          Show
          Allen Wittenauer added a comment - Does the test actually do a DNS lookup?
          Hide
          Jeffrey Naisbitt added a comment -

          According to the java.net.URI javadocs, "No lookup of the host, if any, is performed..." That is the main reason I chose java.net.URI over java.net.URL (even though there are definitely other reasons too).

          Show
          Jeffrey Naisbitt added a comment - According to the java.net.URI javadocs, "No lookup of the host, if any, is performed..." That is the main reason I chose java.net.URI over java.net.URL (even though there are definitely other reasons too).
          Hide
          Jeffrey Naisbitt added a comment -

          'mvn test' passed all the tests except org.apache.hadoop.fs.TestFilterFileSystem, which is a pre-existing failure (unrelated)

          Show
          Jeffrey Naisbitt added a comment - 'mvn test' passed all the tests except org.apache.hadoop.fs.TestFilterFileSystem, which is a pre-existing failure (unrelated)
          Hide
          Jeffrey Naisbitt added a comment -

          It looks like a minor change is needed to make all the mapreduce tests pass.

          Show
          Jeffrey Naisbitt added a comment - It looks like a minor change is needed to make all the mapreduce tests pass.
          Hide
          Tom White added a comment -

          > I didn't see any new problems in the javadoc output, so I'm not sure where the -1 is coming from there. I didn't see anything related to this patch.

          I tried running test-patch.sh on your patch but I couldn't reproduce this (it reported no warnings). Can you have a look in the generated patchJavadocWarnings.txt file for the warnings?

          Show
          Tom White added a comment - > I didn't see any new problems in the javadoc output, so I'm not sure where the -1 is coming from there. I didn't see anything related to this patch. I tried running test-patch.sh on your patch but I couldn't reproduce this (it reported no warnings). Can you have a look in the generated patchJavadocWarnings.txt file for the warnings?
          Hide
          Jeffrey Naisbitt added a comment -

          How are you running test-patch.sh? (what is the actual command you use, and where do you run it?)
          This is my first test-patch run since the mavenization changes, and I am probably doing something wrong since when I run this, it looks like it's basically repeating the expected 6 findbugs warnings 3 times (getting a total of 18). Basically, I'm running this command from my hadoop-trunk folder:
          ./dev-support/test-patch.sh DEVELOPER ../HADOOP-7499.patch /tmp svn grep patch $FINDBUGS_HOME $FORREST_HOME `pwd`

          Show
          Jeffrey Naisbitt added a comment - How are you running test-patch.sh? (what is the actual command you use, and where do you run it?) This is my first test-patch run since the mavenization changes, and I am probably doing something wrong since when I run this, it looks like it's basically repeating the expected 6 findbugs warnings 3 times (getting a total of 18). Basically, I'm running this command from my hadoop-trunk folder: ./dev-support/test-patch.sh DEVELOPER ../ HADOOP-7499 .patch /tmp svn grep patch $FINDBUGS_HOME $FORREST_HOME `pwd`
          Hide
          Tom White added a comment -

          I ran the same command. Can you attach the output warning files please?

          Show
          Tom White added a comment - I ran the same command. Can you attach the output warning files please?
          Hide
          Jeffrey Naisbitt added a comment -

          Here is the /tmp/patchJavadocWarnings.txt file I get.

          Show
          Jeffrey Naisbitt added a comment - Here is the /tmp/patchJavadocWarnings.txt file I get.
          Hide
          Tom White added a comment -

          Thanks Jeffrey. I've created a fix for this in HADOOP-7515.

          Show
          Tom White added a comment - Thanks Jeffrey. I've created a fix for this in HADOOP-7515 .
          Hide
          Jeffrey Naisbitt added a comment -

          Updated patch, fixing an issue with tests failing.

          Show
          Jeffrey Naisbitt added a comment - Updated patch, fixing an issue with tests failing.
          Hide
          Jeffrey Naisbitt added a comment -

          All tests pass on trunk - except for the preexisting org.apache.hadoop.fs.TestFilterFileSystem failure.

          Show
          Jeffrey Naisbitt added a comment - All tests pass on trunk - except for the preexisting org.apache.hadoop.fs.TestFilterFileSystem failure.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12489887/HADOOP-7499-v2.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 6 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build-maven/18//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build-maven/18//artifact/trunk/target/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build-maven/18//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12489887/HADOOP-7499-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build-maven/18//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build-maven/18//artifact/trunk/target/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build-maven/18//console This message is automatically generated.
          Hide
          Jeffrey Naisbitt added a comment -

          So, we discovered that the interface change is incompatible. So, we are removing the incompatible change and the corresponding portion of code relying on that change here and implementing the sanity check only.

          Show
          Jeffrey Naisbitt added a comment - So, we discovered that the interface change is incompatible. So, we are removing the incompatible change and the corresponding portion of code relying on that change here and implementing the sanity check only.
          Hide
          Jeffrey Naisbitt added a comment -

          Updated patches removing the incompatible change and corresponding code references here.

          Passes all tests on hadoop-common.

          trunk common patch test-patch results:
          -1 overall.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 system test framework. The patch passed system test framework compile.

          The findbugs warning is unrelated to this patch.

          Show
          Jeffrey Naisbitt added a comment - Updated patches removing the incompatible change and corresponding code references here. Passes all tests on hadoop-common. trunk common patch test-patch results: -1 overall. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 system test framework. The patch passed system test framework compile. The findbugs warning is unrelated to this patch.
          Hide
          Mahadev konar added a comment -

          +1. Looks good to me! Ill go ahead and push this.

          Show
          Mahadev konar added a comment - +1. Looks good to me! Ill go ahead and push this.
          Hide
          Mahadev konar added a comment -

          I just pushed this. Thanks Jeffrey!

          Show
          Mahadev konar added a comment - I just pushed this. Thanks Jeffrey!
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #727 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/727/)
          HADOOP-7499. Add method for doing a sanity check on hostnames in NetUtils. (Jeffrey Naisbit via mahadev)

          mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1156809
          Files :

          • /hadoop/common/trunk/hadoop-common/src/main/java/org/apache/hadoop/net/CachedDNSToSwitchMapping.java
          • /hadoop/common/trunk/hadoop-common/src/test/java/org/apache/hadoop/net/StaticMapping.java
          • /hadoop/common/trunk/hadoop-common/src/main/java/org/apache/hadoop/net/ScriptBasedMapping.java
          • /hadoop/common/trunk/hadoop-common/src/main/java/org/apache/hadoop/net/DNSToSwitchMapping.java
          • /hadoop/common/trunk/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java
          • /hadoop/common/trunk/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common/src/test/java/org/apache/hadoop/net/TestScriptBasedMapping.java
          • /hadoop/common/trunk/hadoop-common/src/test/java/org/apache/hadoop/net/TestNetUtils.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #727 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/727/ ) HADOOP-7499 . Add method for doing a sanity check on hostnames in NetUtils. (Jeffrey Naisbit via mahadev) mahadev : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1156809 Files : /hadoop/common/trunk/hadoop-common/src/main/java/org/apache/hadoop/net/CachedDNSToSwitchMapping.java /hadoop/common/trunk/hadoop-common/src/test/java/org/apache/hadoop/net/StaticMapping.java /hadoop/common/trunk/hadoop-common/src/main/java/org/apache/hadoop/net/ScriptBasedMapping.java /hadoop/common/trunk/hadoop-common/src/main/java/org/apache/hadoop/net/DNSToSwitchMapping.java /hadoop/common/trunk/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java /hadoop/common/trunk/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common/src/test/java/org/apache/hadoop/net/TestScriptBasedMapping.java /hadoop/common/trunk/hadoop-common/src/test/java/org/apache/hadoop/net/TestNetUtils.java

            People

            • Assignee:
              Jeffrey Naisbitt
              Reporter:
              Jeffrey Naisbitt
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development