Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-491

RAgzip: multiple map tasks for a large gzipped file

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Minor
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      Currently, the hadoop processes gzipped files with only one map.
      We have made a patch that enables multiple map tasks for one large gzipped file. We call the patch RAgzip.
      To process multiple map tasks for gzipped file, you may use RAgzip by just changing InputFormat to RAGZIPInputFormat.
      The option used in RAGZIPInputFormat can be found at the javadoc of RAGZIPInputFormat part.

      RAgzip uses zlib's inflatePrime function which supports random access on a gzipped file.
      Since the inflatePrime is supported from the version of 1.2.2.4, it requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
      RAgzip requires the preprocessing step that creates an access point (.ap) file, which is like the index of the gzipped file chunks.
      The access point(.ap) file is located in same path of the gzipped file.
      If there is a "/user/hadoop/test.gz", the .ap file is created with "/user/hadoop/test.gz.ap".

      Attachments

        1. HADOOP-4652.path
          57 kB
          Daehyun Kim
        2. HADOOP-4652-v2.patch
          58 kB
          Daehyun Kim
        3. HADOOP-4652-v3.patch
          66 kB
          Daehyun Kim
        4. MAPREDUCE-491.patch
          14 kB
          Daehyun Kim

        Issue Links

          Activity

            People

              ohyun49kdh Daehyun Kim
              ohyun49kdh Daehyun Kim
              Votes:
              0 Vote for this issue
              Watchers:
              20 Start watching this issue

              Dates

                Created:
                Updated: