Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.19.0
    • Fix Version/s: 0.19.0
    • Component/s: conf, io
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Introduced support for bzip2 compressed files.

      Description

      Hadoop recognizes gzip compressed input and automatically decompresses the data before providing it to the mapper. But Hadoop can not split a gzip stream due to the very nature of the gzip compression. Consequently one gzip stream (e.g a whole file) can go to only one mapper. On the contrary Bzip2 compressed stream can be split across its block delimiters.

      We are interested in extending Hadoop to support splittable bzip2 with a codec. (https://issues.apache.org/jira/browse/HADOOP-1823 uses input reader to split the bzip2 files, which must be provided by the user and can handle FileInputFormat. If a user wants to use some other input format or wants to do custom record handling, he must write a new input reader!)

      We have a patch now that provides a basic bzip2 codec equivalent to the current gzip codec. We are in the process of extending that to support splitting.

      1. HADOOP-3646-version5.patch
        108 kB
        Abdul Qadeer
      2. HADOOP-3646-version4.patch
        109 kB
        Abdul Qadeer
      3. HADOOP-3646version3.patch
        107 kB
        Abdul Qadeer
      4. HADOOP-3646.patch
        130 kB
        Abdul Qadeer
      5. HADOOP-3646.patch
        134 kB
        Abdul Qadeer

        Issue Links

          Activity

            People

            • Assignee:
              Abdul Qadeer
              Reporter:
              Abdul Qadeer
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 1,008h
                1,008h
                Remaining:
                Remaining Estimate - 1,008h
                1,008h
                Logged:
                Time Spent - Not Specified
                Not Specified

                  Development