Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-4012

Providing splitting support for bzip2 compressed files

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: io
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      BZip2 files can now be split.

      Description

      Hadoop assumes that if the input data is compressed, it can not be split (mainly due to the limitation of many codecs that they need the whole input stream to decompress successfully). So in such a case, Hadoop prepares only one split per compressed file, where the lower split limit is at 0 while the upper limit is the end of the file. The consequence of this decision is that, one compress file goes to a single mapper. Although it circumvents the limitation of codecs (as mentioned above) but reduces the parallelism substantially, as it was possible otherwise in case of splitting.

      BZip2 is a compression / De-Compression algorithm which does compression on blocks of data and later these compressed blocks can be decompressed independent of each other. This is indeed an opportunity that instead of one BZip2 compressed file going to one mapper, we can process chunks of file in parallel. The correctness criteria of such a processing is that for a bzip2 compressed file, each compressed block should be processed by only one mapper and ultimately all the blocks of the file should be processed. (By processing we mean the actual utilization of that un-compressed data (coming out of the codecs) in a mapper).

      We are writing the code to implement this suggested functionality. Although we have used bzip2 as an example, but we have tried to extend Hadoop's compression interfaces so that any other codecs with the same capability as that of bzip2, could easily use the splitting support. The details of these changes will be posted when we submit the code.

        Attachments

        1. C4012-14.patch
          61 kB
          Chris Douglas
        2. C4012-13.patch
          52 kB
          Abdul Qadeer
        3. C4012-12.patch
          55 kB
          Chris Douglas
        4. Hadoop-4012-version11.patch
          51 kB
          Abdul Qadeer
        5. Hadoop-4012-version10.patch
          51 kB
          Abdul Qadeer
        6. Hadoop-4012-version9.patch
          70 kB
          Abdul Qadeer
        7. Hadoop-4012-version8.patch
          70 kB
          Abdul Qadeer
        8. Hadoop-4012-version7.patch
          67 kB
          Abdul Qadeer
        9. Hadoop-4012-version6.patch
          51 kB
          Abdul Qadeer
        10. Hadoop-4012-version5.patch
          51 kB
          Abdul Qadeer
        11. Hadoop-4012-version4.patch
          52 kB
          Abdul Qadeer
        12. Hadoop-4012-version3.patch
          52 kB
          Abdul Qadeer
        13. Hadoop-4012-version2.patch
          52 kB
          Abdul Qadeer
        14. Hadoop-4012-version1.patch
          51 kB
          Abdul Qadeer

          Issue Links

            Activity

              People

              • Assignee:
                aqadeer Abdul Qadeer
                Reporter:
                aqadeer Abdul Qadeer
              • Votes:
                2 Vote for this issue
                Watchers:
                23 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: