Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.19.0
    • Component/s: conf, io
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Introduced support for bzip2 compressed files.

      Description

      Hadoop recognizes gzip compressed input and automatically decompresses the data before providing it to the mapper. But Hadoop can not split a gzip stream due to the very nature of the gzip compression. Consequently one gzip stream (e.g a whole file) can go to only one mapper. On the contrary Bzip2 compressed stream can be split across its block delimiters.

      We are interested in extending Hadoop to support splittable bzip2 with a codec. (https://issues.apache.org/jira/browse/HADOOP-1823 uses input reader to split the bzip2 files, which must be provided by the user and can handle FileInputFormat. If a user wants to use some other input format or wants to do custom record handling, he must write a new input reader!)

      We have a patch now that provides a basic bzip2 codec equivalent to the current gzip codec. We are in the process of extending that to support splitting.

        Attachments

        1. HADOOP-3646-version5.patch
          108 kB
          Abdul Qadeer
        2. HADOOP-3646-version4.patch
          109 kB
          Abdul Qadeer
        3. HADOOP-3646version3.patch
          107 kB
          Abdul Qadeer
        4. HADOOP-3646.patch
          130 kB
          Abdul Qadeer
        5. HADOOP-3646.patch
          134 kB
          Abdul Qadeer

          Issue Links

            Activity

              People

              • Assignee:
                aqadeer Abdul Qadeer
                Reporter:
                aqadeer Abdul Qadeer
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - 1,008h
                  1,008h
                  Remaining:
                  Remaining Estimate - 1,008h
                  1,008h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified