Hadoop Common
  1. Hadoop Common
  2. HADOOP-8390

TestFileSystemCanonicalization fails with JDK7

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: test
    • Labels:
    • Environment:

      Description

      Failed tests:
      testShortAuthority(org.apache.hadoop.fs.TestFileSystemCanonicalization):
      expected:<myfs://host.a.b:123> but was:<myfs://host:123>
      testPartialAuthority(org.apache.hadoop.fs.TestFileSystemCanonicalization):
      expected:<myfs://host.a.b:123> but was:<myfs://host.a:123>

      Passes on same machine with JDK 1.6.0_32.

      1. HADOOP-8390.patch
        0.7 kB
        Trevor Robinson
      2. HADOOP-8390-BeforeClass.patch
        1 kB
        Trevor Robinson

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #1169 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1169/)
          HADOOP-8390. TestFileSystemCanonicalization fails with JDK7 (Trevor Robinson via tgraves) (Revision 1374084)

          Result = FAILURE
          tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374084
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCanonicalization.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1169 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1169/ ) HADOOP-8390 . TestFileSystemCanonicalization fails with JDK7 (Trevor Robinson via tgraves) (Revision 1374084) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374084 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCanonicalization.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #1137 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1137/)
          HADOOP-8390. TestFileSystemCanonicalization fails with JDK7 (Trevor Robinson via tgraves) (Revision 1374084)

          Result = FAILURE
          tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374084
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCanonicalization.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1137 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1137/ ) HADOOP-8390 . TestFileSystemCanonicalization fails with JDK7 (Trevor Robinson via tgraves) (Revision 1374084) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374084 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCanonicalization.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-0.23-Build #346 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/346/)
          merge -r 1374084:1374085 from branch-2. FIXES: HADOOP-8390 (Revision 1374088)

          Result = SUCCESS
          tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374088
          Files :

          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCanonicalization.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #346 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/346/ ) merge -r 1374084:1374085 from branch-2. FIXES: HADOOP-8390 (Revision 1374088) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374088 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCanonicalization.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #2590 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2590/)
          HADOOP-8390. TestFileSystemCanonicalization fails with JDK7 (Trevor Robinson via tgraves) (Revision 1374084)

          Result = SUCCESS
          tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374084
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCanonicalization.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2590 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2590/ ) HADOOP-8390 . TestFileSystemCanonicalization fails with JDK7 (Trevor Robinson via tgraves) (Revision 1374084) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374084 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCanonicalization.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #2655 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2655/)
          HADOOP-8390. TestFileSystemCanonicalization fails with JDK7 (Trevor Robinson via tgraves) (Revision 1374084)

          Result = SUCCESS
          tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374084
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCanonicalization.java
          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2655 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2655/ ) HADOOP-8390 . TestFileSystemCanonicalization fails with JDK7 (Trevor Robinson via tgraves) (Revision 1374084) Result = SUCCESS tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374084 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCanonicalization.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #2620 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2620/)
          HADOOP-8390. TestFileSystemCanonicalization fails with JDK7 (Trevor Robinson via tgraves) (Revision 1374084)

          Result = FAILURE
          tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374084
          Files :

          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
          • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCanonicalization.java
          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2620 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2620/ ) HADOOP-8390 . TestFileSystemCanonicalization fails with JDK7 (Trevor Robinson via tgraves) (Revision 1374084) Result = FAILURE tgraves : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1374084 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/TestFileSystemCanonicalization.java
          Hide
          Thomas Graves added a comment -

          +1. Thanks Trevor! I'll commit this shortly.

          Show
          Thomas Graves added a comment - +1. Thanks Trevor! I'll commit this shortly.
          Hide
          Trevor Robinson added a comment -

          Thanks. Could you commit this and/or review the other JDK7 fixes?

          Show
          Trevor Robinson added a comment - Thanks. Could you commit this and/or review the other JDK7 fixes ?
          Hide
          Daryn Sharp added a comment -

          +1 Good job tracking down the root cause!

          Show
          Daryn Sharp added a comment - +1 Good job tracking down the root cause!
          Hide
          Trevor Robinson added a comment -

          Test failures are HADOOP-8699

          Show
          Trevor Robinson added a comment - Test failures are HADOOP-8699
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12540917/HADOOP-8390-BeforeClass.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

          org.apache.hadoop.fs.TestS3_LocalFileContextURI
          org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract
          org.apache.hadoop.fs.TestLocal_S3FileContextURI
          org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1293//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1293//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12540917/HADOOP-8390-BeforeClass.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.fs.TestS3_LocalFileContextURI org.apache.hadoop.fs.s3native.TestInMemoryNativeS3FileSystemContract org.apache.hadoop.fs.TestLocal_S3FileContextURI org.apache.hadoop.fs.s3.TestInMemoryS3FileSystemContract +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1293//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1293//console This message is automatically generated.
          Hide
          Trevor Robinson added a comment -

          That's a larger change because @BeforeClass only works if the class doesn't extend TestCase: http://stackoverflow.com/questions/733037/why-isnt-my-beforeclass-method-running

          Still, it's probably better to upgrade to modern, annotation-based tests, so I'll attach a patch.

          Show
          Trevor Robinson added a comment - That's a larger change because @BeforeClass only works if the class doesn't extend TestCase: http://stackoverflow.com/questions/733037/why-isnt-my-beforeclass-method-running Still, it's probably better to upgrade to modern, annotation-based tests, so I'll attach a patch.
          Hide
          Daryn Sharp added a comment -

          Instead of @Test, should it maybe be @BeforeClass?

          Show
          Daryn Sharp added a comment - Instead of @Test , should it maybe be @BeforeClass ?
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12540782/HADOOP-8390.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1289//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1289//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12540782/HADOOP-8390.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1289//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1289//console This message is automatically generated.
          Hide
          Trevor Robinson added a comment -

          Confirmation of the JDK7 issue: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7023180

          Starting in build 129 of JDK 7, the order of methods returned by getDeclaredMethods changed and can vary from run to run. This has been observed to cause issues for applications relying on the specified-to-be-unspecified ordering of methods retuned by getDeclaredMethods.
          The previously implementation of getDeclaredMethods did not have a firm ordering guarantee and the specification does not require one. Merely returning a consistent order throughout the run of a VM would not be sufficient to address programs expecting a (mostly) sorted order.
          Imposing a predictable ordering is not being considered at this time; closing as not a bug.

          Show
          Trevor Robinson added a comment - Confirmation of the JDK7 issue: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7023180 Starting in build 129 of JDK 7, the order of methods returned by getDeclaredMethods changed and can vary from run to run. This has been observed to cause issues for applications relying on the specified-to-be-unspecified ordering of methods retuned by getDeclaredMethods. The previously implementation of getDeclaredMethods did not have a firm ordering guarantee and the specification does not require one. Merely returning a consistent order throughout the run of a VM would not be sufficient to address programs expecting a (mostly) sorted order. Imposing a predictable ordering is not being considered at this time; closing as not a bug.
          Hide
          Trevor Robinson added a comment -

          Err, JDK7 doesn't seem to preserve the declaration order of class members for reflection.

          The reflection methods warn about order being undefined, but JDK6 seemed to preserve it. testSetupResolver() was declared first, so Junit with JDK6 ran it first.

          Show
          Trevor Robinson added a comment - Err, JDK7 doesn't seem to preserve the declaration order of class members for reflection. The reflection methods warn about order being undefined, but JDK6 seemed to preserve it. testSetupResolver() was declared first, so Junit with JDK6 ran it first.
          Hide
          Trevor Robinson added a comment -

          This is simply a test order-dependency bug. testSetupResolver() is declared as a @Test, but just performs static initialization required by most of the other tests (NetUtilsTestResolver.install()). The attached patch changes this test method to a static initializer block.

          Perhaps the reason this breaks with JDK7 is that it doesn't seem to preserve the declaration of class members for reflection.

          Show
          Trevor Robinson added a comment - This is simply a test order-dependency bug. testSetupResolver() is declared as a @Test , but just performs static initialization required by most of the other tests ( NetUtilsTestResolver.install() ). The attached patch changes this test method to a static initializer block. Perhaps the reason this breaks with JDK7 is that it doesn't seem to preserve the declaration of class members for reflection.
          Hide
          Eric Charles added a comment -

          Issue confirmed here on ubuntu: failure on jdk7, success on jdk6.
          See also HADOOP-8577 where @HarshJ mentions a fix needed on /etc/hosts - Don't know if setting a proper /etc/hosts will help?

          Show
          Eric Charles added a comment - Issue confirmed here on ubuntu: failure on jdk7, success on jdk6. See also HADOOP-8577 where @HarshJ mentions a fix needed on /etc/hosts - Don't know if setting a proper /etc/hosts will help?
          Hide
          Daryn Sharp added a comment -

          This is not good... Host-based tokens and fs canonicalization for the tokens depend on correct behavior. I've invested a lot in those features, so please be sure I have a chance to review the patch.

          Show
          Daryn Sharp added a comment - This is not good... Host-based tokens and fs canonicalization for the tokens depend on correct behavior. I've invested a lot in those features, so please be sure I have a chance to review the patch.

            People

            • Assignee:
              Trevor Robinson
              Reporter:
              Trevor Robinson
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development