Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1927

unit test for HADOOP-6835 (concatenated gzip support)

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.23.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      More extensive test of concatenated gzip (and bzip2) decoding support for HADOOP-6835 (and HADOOP-4012 and HADOOP-6852).

        Activity

        Hide
        Greg Roelofs added a comment -

        Test code, six associated test files, and build.xml changes to copy them to the appropriate location.

        Show
        Greg Roelofs added a comment - Test code, six associated test files, and build.xml changes to copy them to the appropriate location.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12449019/MR-1927.v9.trunk-hadoop-mapreduce.patch
        against trunk revision 961578.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 21 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/294/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/294/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/294/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/294/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12449019/MR-1927.v9.trunk-hadoop-mapreduce.patch against trunk revision 961578. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 21 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/294/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/294/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/294/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/294/console This message is automatically generated.
        Hide
        Greg Roelofs added a comment -

        Updated, test-patch-passing patch coming shortly.

        Show
        Greg Roelofs added a comment - Updated, test-patch-passing patch coming shortly.
        Hide
        Greg Roelofs added a comment -

        Script and gzip component subfiles to generate all 32 unencrypted flagbits variants (that is, text/binary, comment/no comment, header CRC/no header CRC, etc.).

        These are not currently used by the unit test, but they could be. Is it worth it?

        (Note that gzip versions up through 1.4 cannot be used to test the 16 header-CRC variants. A fix was checked into gzip's trunk on 20100703, however: http://git.savannah.gnu.org/cgit/gzip.git/commit/?id=97e225c0e8ee24f3fc9ff7c5cec63e74aefbb6a0 . zlib-based gzip decoders [zlib >= 1.2.1.2] can read all 32 variants successfully.)

        Show
        Greg Roelofs added a comment - Script and gzip component subfiles to generate all 32 unencrypted flagbits variants (that is, text/binary, comment/no comment, header CRC/no header CRC, etc.). These are not currently used by the unit test, but they could be. Is it worth it? (Note that gzip versions up through 1.4 cannot be used to test the 16 header-CRC variants. A fix was checked into gzip's trunk on 20100703, however: http://git.savannah.gnu.org/cgit/gzip.git/commit/?id=97e225c0e8ee24f3fc9ff7c5cec63e74aefbb6a0 . zlib-based gzip decoders [zlib >= 1.2.1.2] can read all 32 variants successfully.)
        Hide
        Greg Roelofs added a comment -

        Oops, small correction: zlib must be newer than 1.2.1.2 to handle header CRCs correctly. RHEL4 ships with 1.2.1.2, and it's buggy.

        Show
        Greg Roelofs added a comment - Oops, small correction: zlib must be newer than 1.2.1.2 to handle header CRCs correctly. RHEL4 ships with 1.2.1.2, and it's buggy.
        Hide
        Greg Roelofs added a comment -

        Updated (final?) patch. This corrects the internals of the gzip extra field in one of the test files (not relevant to the test, but matter of spec-compliance), changes a native-libs-only assertion to a warning to work around Apache Hudson's lack of support, and removes all but three commented-out code blocks.

        The three remaining blocks are all a result of other bugs: the aforementioned Hudson native-library one (lack of coverage of a key infrastructure component is a problem, and it's getting worse with every new native codec); HADOOP-6799 (broken reset() on gzip compression); and HADOOP-6852 (broken bzip2 decompression, apparently). I realize commented-out code is frowned upon as a style issue, but I would rather not remove useful tests that are currently blocked by things outside my (immediate) control. I'll defer to committer's opinion, however.

        This passed "ant test-patch" on a 10-node cluster, so I'm going to "submit patch" to see if it gets past Apache's Hudson, too.

        Show
        Greg Roelofs added a comment - Updated (final?) patch. This corrects the internals of the gzip extra field in one of the test files (not relevant to the test, but matter of spec-compliance), changes a native-libs-only assertion to a warning to work around Apache Hudson's lack of support, and removes all but three commented-out code blocks. The three remaining blocks are all a result of other bugs: the aforementioned Hudson native-library one (lack of coverage of a key infrastructure component is a problem, and it's getting worse with every new native codec); HADOOP-6799 (broken reset() on gzip compression); and HADOOP-6852 (broken bzip2 decompression, apparently). I realize commented-out code is frowned upon as a style issue, but I would rather not remove useful tests that are currently blocked by things outside my (immediate) control. I'll defer to committer's opinion, however. This passed "ant test-patch" on a 10-node cluster, so I'm going to "submit patch" to see if it gets past Apache's Hudson, too.
        Hide
        Greg Roelofs added a comment -

        Extended hadoop-mapreduce unit test for hadoop-common's gzip and bzip2 codecs. (Supplements TestCodec in hadoop-common.)

        Show
        Greg Roelofs added a comment - Extended hadoop-mapreduce unit test for hadoop-common's gzip and bzip2 codecs. (Supplements TestCodec in hadoop-common.)
        Hide
        Greg Roelofs added a comment -

        Looks like Hudson's gone off the deep end again. Here's my "ant test-patch" results:

        [exec] +1 overall.
        [exec]
        [exec] +1 @author. The patch does not contain any @author tags.
        [exec]
        [exec] +1 tests included. The patch appears to include 21 new or modified tests.
        [exec]
        [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
        [exec]
        [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
        [exec]
        [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
        [exec]
        [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

        BUILD SUCCESSFUL
        Total time: 39 minutes 15 seconds

        Show
        Greg Roelofs added a comment - Looks like Hudson's gone off the deep end again. Here's my "ant test-patch" results: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 21 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. BUILD SUCCESSFUL Total time: 39 minutes 15 seconds
        Hide
        Greg Roelofs added a comment -

        Fixed the internal structure of the extra field in one of the test files (not relevant to the test, but a matter of spec correctness), as well as the corresponding file length in the unit test (3409 -> 3413 bytes). No substantive changes since v12.

        Show
        Greg Roelofs added a comment - Fixed the internal structure of the extra field in one of the test files (not relevant to the test, but a matter of spec correctness), as well as the corresponding file length in the unit test (3409 -> 3413 bytes). No substantive changes since v12.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12452999/MR-1927.v13.trunk-hadoop-mapreduce.patch.txt
        against trunk revision 1074251.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 21 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        -1 contrib tests. The patch failed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/75//testReport/
        Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/75//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/75//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12452999/MR-1927.v13.trunk-hadoop-mapreduce.patch.txt against trunk revision 1074251. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 21 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/75//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/75//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/75//console This message is automatically generated.
        Hide
        Tom White added a comment -

        +1 I just committed this. Thanks Greg.

        Show
        Tom White added a comment - +1 I just committed this. Thanks Greg.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #616 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/616/)
        MAPREDUCE-1927. Unit test for HADOOP-6835 (concatenated gzip support). Contributed by Greg Roelofs.

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #616 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/616/ ) MAPREDUCE-1927 . Unit test for HADOOP-6835 (concatenated gzip support). Contributed by Greg Roelofs.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )

          People

          • Assignee:
            Greg Roelofs
            Reporter:
            Greg Roelofs
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development