Hadoop Common
  1. Hadoop Common
  2. HADOOP-7111

Several TFile tests failing when native libraries are present

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.23.0
    • Component/s: io
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When running tests with native libraries present, TestTFileByteArrays and TestTFileJClassComparatorByteArrays fail on trunk. They don't seem to fail in 0.20 with native libraries.

      1. hadoop-7111.1.patch
        25 kB
        Aaron T. Myers
      2. hadoop-7111.0.patch
        25 kB
        Aaron T. Myers

        Issue Links

          Activity

          Todd Lipcon created issue -
          Todd Lipcon made changes -
          Field Original Value New Value
          Link This issue is broken by HADOOP-6683 [ HADOOP-6683 ]
          Hide
          Todd Lipcon added a comment -

          This is a regression caused by HADOOP-6683. Reverting that patch allows the test to patch.

          Haven't looked into it yet whether the tests are faulty or that patch is faulty.

          Show
          Todd Lipcon added a comment - This is a regression caused by HADOOP-6683 . Reverting that patch allows the test to patch. Haven't looked into it yet whether the tests are faulty or that patch is faulty.
          Hide
          Hong Tang added a comment -

          There could be two possibilities.

          1. in some of the TFile tests, we hard-coded the sizes of the compressed blocks. HADOOP-6683 may have affected the compressed sizes.
          2. TFile closes a block when it observes the compressed size of the block exceeds the configured minimum block size. HADOOP-6683 waits until a full-buffer if raw bytes before compressing, and thus changes the behavior.
          Show
          Hong Tang added a comment - There could be two possibilities. in some of the TFile tests, we hard-coded the sizes of the compressed blocks. HADOOP-6683 may have affected the compressed sizes. TFile closes a block when it observes the compressed size of the block exceeds the configured minimum block size. HADOOP-6683 waits until a full-buffer if raw bytes before compressing, and thus changes the behavior.
          Hide
          Todd Lipcon added a comment -

          Something in trunk just changed such that the native libs are on the classpath when the tests run, so the tests are now failing:

          https://builds.apache.org/job/Hadoop-Common-trunk/721/testReport/junit/org.apache.hadoop.io.file.tfile/TestTFileByteArrays/testOneBlockPlusOneEntry/

          Show
          Todd Lipcon added a comment - Something in trunk just changed such that the native libs are on the classpath when the tests run, so the tests are now failing: https://builds.apache.org/job/Hadoop-Common-trunk/721/testReport/junit/org.apache.hadoop.io.file.tfile/TestTFileByteArrays/testOneBlockPlusOneEntry/
          Todd Lipcon made changes -
          Link This issue is related to HADOOP-7409 [ HADOOP-7409 ]
          Hide
          Eli Collins added a comment -

          Strange, bi-secting on my host leads to HADOOP-7206 but that was committed after these tests started failing, must be triggering something.

          Show
          Eli Collins added a comment - Strange, bi-secting on my host leads to HADOOP-7206 but that was committed after these tests started failing, must be triggering something.
          Todd Lipcon made changes -
          Link This issue is duplicated by HADOOP-7453 [ HADOOP-7453 ]
          Aaron T. Myers made changes -
          Assignee Aaron T. Myers [ atm ]
          Hide
          Aaron T. Myers added a comment -

          Patch which addresses the issue. Hong's theory #2 was correct.

          The only change which actually fixes the tests is the change to the initialization of records1stBlock and records2ndBlock to be dependent upon whether or not the native libs are present.

          While I was in the file, I also took the opportunity to do a little cleanup by:

          1. Converting to the new style of test annotations.
          2. Refactoring some methods which had unused parameters.
          3. Fixing up some formatting.
          4. Reducing some code duplication in TestTFileJClassComparatorByteArrays
          5. Fixing some "unused variable" compiler warnings.

          I've tested both TestTFileJClassComparatorByteArrays and TestTFileByteArrays both with and without native code being present.

          Show
          Aaron T. Myers added a comment - Patch which addresses the issue. Hong's theory #2 was correct. The only change which actually fixes the tests is the change to the initialization of records1stBlock and records2ndBlock to be dependent upon whether or not the native libs are present. While I was in the file, I also took the opportunity to do a little cleanup by: Converting to the new style of test annotations. Refactoring some methods which had unused parameters. Fixing up some formatting. Reducing some code duplication in TestTFileJClassComparatorByteArrays Fixing some "unused variable" compiler warnings. I've tested both TestTFileJClassComparatorByteArrays and TestTFileByteArrays both with and without native code being present.
          Aaron T. Myers made changes -
          Attachment hadoop-7111.0.patch [ 12487068 ]
          Aaron T. Myers made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12487068/hadoop-7111.0.patch
          against trunk revision 1147971.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/750//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/750//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/750//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12487068/hadoop-7111.0.patch against trunk revision 1147971. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/750//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/750//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/750//console This message is automatically generated.
          Aaron T. Myers made changes -
          Fix Version/s 0.23.0 [ 12315569 ]
          Fix Version/s 0.22.0 [ 12314296 ]
          Hide
          Tom White added a comment -

          +1

          The TODO you added should be removed along with the assertNull on the following line, since the Assert.fail is sufficient.

          Show
          Tom White added a comment - +1 The TODO you added should be removed along with the assertNull on the following line, since the Assert.fail is sufficient.
          Hide
          Aaron T. Myers added a comment -

          Thanks a lot, Tom. Here's a patch removing the TODO and the useless assert.

          Show
          Aaron T. Myers added a comment - Thanks a lot, Tom. Here's a patch removing the TODO and the useless assert.
          Aaron T. Myers made changes -
          Attachment hadoop-7111.1.patch [ 12487211 ]
          Hide
          Aaron T. Myers added a comment -

          I've just committed this. Looking forward to a green Common build!

          Thanks again for the review, Tom.

          Show
          Aaron T. Myers added a comment - I've just committed this. Looking forward to a green Common build! Thanks again for the review, Tom.
          Aaron T. Myers made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12487211/hadoop-7111.1.patch
          against trunk revision 1147971.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 9 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 system test framework. The patch passed system test framework compile.

          Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/754//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/754//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/754//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12487211/hadoop-7111.1.patch against trunk revision 1147971. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/754//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HADOOP-Build/754//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/754//console This message is automatically generated.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #695 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/695/)
          HADOOP-7111. Several TFile tests failing when native libraries are present. (atm)

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1148933
          Files :

          • /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileStreams.java
          • /hadoop/common/trunk/common/CHANGES.txt
          • /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileJClassComparatorByteArrays.java
          • /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileByteArrays.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #695 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/695/ ) HADOOP-7111 . Several TFile tests failing when native libraries are present. (atm) atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1148933 Files : /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileStreams.java /hadoop/common/trunk/common/CHANGES.txt /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileJClassComparatorByteArrays.java /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileByteArrays.java
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #753 (See https://builds.apache.org/job/Hadoop-Common-trunk/753/)
          HADOOP-7111. Several TFile tests failing when native libraries are present. (atm)

          atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1148933
          Files :

          • /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileStreams.java
          • /hadoop/common/trunk/common/CHANGES.txt
          • /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileJClassComparatorByteArrays.java
          • /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileByteArrays.java
          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #753 (See https://builds.apache.org/job/Hadoop-Common-trunk/753/ ) HADOOP-7111 . Several TFile tests failing when native libraries are present. (atm) atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1148933 Files : /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileStreams.java /hadoop/common/trunk/common/CHANGES.txt /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileJClassComparatorByteArrays.java /hadoop/common/trunk/common/src/test/core/org/apache/hadoop/io/file/tfile/TestTFileByteArrays.java
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Todd Lipcon added a comment -

          These seem to be failing again in some patch builds. I filed HADOOP-8537.

          Show
          Todd Lipcon added a comment - These seem to be failing again in some patch builds. I filed HADOOP-8537 .
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          181d 23h 25m 1 Aaron T. Myers 19/Jul/11 23:22
          Patch Available Patch Available Resolved Resolved
          22h 20m 1 Aaron T. Myers 20/Jul/11 21:43
          Resolved Resolved Closed Closed
          117d 4h 7m 1 Arun C Murthy 15/Nov/11 00:50

            People

            • Assignee:
              Aaron T. Myers
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development