Hadoop Common
  1. Hadoop Common
  2. HADOOP-7111

Several TFile tests failing when native libraries are present

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: io
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.

      1. hadoop-7111.0.patch
        25 kB
        Aaron T. Myers
      2. hadoop-7111.1.patch
        25 kB
        Aaron T. Myers

        Issue Links

          Activity

          Hide
          Todd Lipcon added a comment -

          These seem to be failing again in some patch builds. I filed HADOOP-8537.

          Show
          Todd Lipcon added a comment - These seem to be failing again in some patch builds. I filed HADOOP-8537 .
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #753 (See https://builds.apache.org/job/Hadoop-Common-trunk/753/)
          HADOOP-7111. Several TFile tests failing when native libraries are present. (atm)

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1148933
          Files :

          • /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileStreams.java
          • /hadoop/common/trunk/common/CHANGES.txt
          • /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileJClassComparatorByteArrays.java
          • /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileByteArrays.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #753 (See https://builds.apache.org/job/Hadoop-Common-trunk/753/ ) HADOOP-7111 . Several TFile tests failing when native libraries are present. (atm) atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1148933 Files : /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileStreams.java /hadoop/common/trunk/common/CHANGES.txt /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileJClassComparatorByteArrays.java /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileByteArrays.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #695 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/695/)
          HADOOP-7111. Several TFile tests failing when native libraries are present. (atm)

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1148933
          Files :

          • /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileStreams.java
          • /hadoop/common/trunk/common/CHANGES.txt
          • /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileJClassComparatorByteArrays.java
          • /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileByteArrays.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #695 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/695/ ) HADOOP-7111 . Several TFile tests failing when native libraries are present. (atm) atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1148933 Files : /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileStreams.java /hadoop/common/trunk/common/CHANGES.txt /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileJClassComparatorByteArrays.java /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileByteArrays.java
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12487211/hadoop-7111.1.patch
          against trunk revision 1147971.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/754//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/754//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/754//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12487211/hadoop-7111.1.patch against trunk revision 1147971. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/754//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/754//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/754//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          I've just committed this. Looking forward to a green Common build!

          Thanks again for the review, Tom.

          Show
          Aaron T. Myers added a comment - I've just committed this. Looking forward to a green Common build! Thanks again for the review, Tom.
          Hide
          Aaron T. Myers added a comment -

          Thanks a lot, Tom. Here's a patch removing the TODO and the useless assert.

          Show
          Aaron T. Myers added a comment - Thanks a lot, Tom. Here's a patch removing the TODO and the useless assert.
          Hide
          Tom White added a comment -

          +1

          The TODO you added should be removed along with the assertNull on the following line, since the Assert.fail is sufficient.

          Show
          Tom White added a comment - +1 The TODO you added should be removed along with the assertNull on the following line, since the Assert.fail is sufficient.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12487068/hadoop-7111.0.patch
          against trunk revision 1147971.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/750//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/750//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/750//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12487068/hadoop-7111.0.patch against trunk revision 1147971. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/750//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/750//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/750//console This message is automatically generated.
          Hide
          Aaron T. Myers added a comment -

          Patch which addresses the issue. Hong's theory #2 was correct.

          The only change which actually fixes the tests is the change to the initialization of records1stBlock and records2ndBlock to be dependent upon whether or not the native libs are present.

          While I was in the file, I also took the opportunity to do a little cleanup by:

          1. Converting to the new style of test annotations.
          2. Refactoring some methods which had unused parameters.
          3. Fixing up some formatting.
          4. Reducing some code duplication in TestTFileJClassComparatorByteArrays
          5. Fixing some "unused variable" compiler warnings.

          I've tested both TestTFileJClassComparatorByteArrays and TestTFileByteArrays both with and without native code being present.

          Show
          Aaron T. Myers added a comment - Patch which addresses the issue. Hong's theory #2 was correct. The only change which actually fixes the tests is the change to the initialization of records1stBlock and records2ndBlock to be dependent upon whether or not the native libs are present. While I was in the file, I also took the opportunity to do a little cleanup by: Converting to the new style of test annotations. Refactoring some methods which had unused parameters. Fixing up some formatting. Reducing some code duplication in TestTFileJClassComparatorByteArrays Fixing some "unused variable" compiler warnings. I've tested both TestTFileJClassComparatorByteArrays and TestTFileByteArrays both with and without native code being present.
          Hide
          Eli Collins added a comment -

          Strange, bi-secting on my host leads to HADOOP-7206 but that was committed after these tests started failing, must be triggering something.

          Show
          Eli Collins added a comment - Strange, bi-secting on my host leads to HADOOP-7206 but that was committed after these tests started failing, must be triggering something.
          Hide
          Todd Lipcon added a comment -

          Something in trunk just changed such that the native libs are on the classpath when the tests run, so the tests are now failing:

          https://builds.apache.org/job/Hadoop-Common-trunk/721/testReport/junit/org.apache.hadoop.io.file.tfile/TestTFileByteArrays/testOneBlockPlusOneEntry/

          Show
          Todd Lipcon added a comment - Something in trunk just changed such that the native libs are on the classpath when the tests run, so the tests are now failing: https://builds.apache.org/job/Hadoop-Common-trunk/721/testReport/junit/org.apache.hadoop.io.file.tfile/TestTFileByteArrays/testOneBlockPlusOneEntry/
          Hide
          Hong Tang added a comment -

          There could be two possibilities.

          1. in some of the TFile tests, we hard-coded the sizes of the compressed blocks. HADOOP-6683 may have affected the compressed sizes.
          2. TFile closes a block when it observes the compressed size of the block exceeds the configured minimum block size. HADOOP-6683 waits until a full-buffer if raw bytes before compressing, and thus changes the behavior.
          Show
          Hong Tang added a comment - There could be two possibilities. in some of the TFile tests, we hard-coded the sizes of the compressed blocks. HADOOP-6683 may have affected the compressed sizes. TFile closes a block when it observes the compressed size of the block exceeds the configured minimum block size. HADOOP-6683 waits until a full-buffer if raw bytes before compressing, and thus changes the behavior.
          Hide
          Todd Lipcon added a comment -

          This is a regression caused by HADOOP-6683. Reverting that patch allows the test to patch.

          Haven't looked into it yet whether the tests are faulty or that patch is faulty.

          Show
          Todd Lipcon added a comment - This is a regression caused by HADOOP-6683 . Reverting that patch allows the test to patch. Haven't looked into it yet whether the tests are faulty or that patch is faulty.

            People

            • Assignee:
              Aaron T. Myers
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development