Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-1

Switch lzo to other compression algorithm for Chukwa Records

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Redhat 5.1, Java 6

      Description

      Chukwa Records depends on the lzo library from Hadoop. Hadoop is removing lzo, therefore Chukwa needs to switch to another compression algorithm. This jira is to keep track of the shell script and java file changes.

      1. CHUKWA-1.patch
        3 kB
        Jerome Boulon

        Issue Links

          Activity

          Hide
          jboulon Jerome Boulon added a comment -

          remove hard-coded reference to LZO

          Show
          jboulon Jerome Boulon added a comment - remove hard-coded reference to LZO
          Hide
          jboulon Jerome Boulon added a comment -

          To be compliant with Apache license we have to remove all references to LZO, however you can still use LZO and achieve the same behavior by adding the lines below to your hadoop-site.xml or chukwa-demux-conf.xml.

          <property>
          <name>mapred.output.compress</name>
          <value>true</value>
          <description>Should the job outputs be compressed?
          </description>
          </property>

          <property>
          <name>mapred.output.compression.type</name>
          <value>BLOCK</value>
          <description>If the job outputs are to compressed as SequenceFiles, how should
          they be compressed? Should be one of NONE, RECORD or BLOCK.
          </description>
          </property>

          <property>
          <name>mapred.output.compression.codec</name>
          <value>org.apache.hadoop.io.compress.LzoCodec</value>
          <description>If the job outputs are compressed, how should they be compressed?
          </description>
          </property>

          <property>
          <name>mapred.compress.map.output</name>
          <value>true</value>
          <description>Should the outputs of the maps be compressed before being
          sent across the network. Uses SequenceFile compression.
          </description>
          </property>

          <property>
          <name>mapred.map.output.compression.codec</name>
          <value>org.apache.hadoop.io.compress.LzoCodec</value>
          <description>If the map outputs are compressed, how should they be
          compressed?
          </description>
          </property>

          Show
          jboulon Jerome Boulon added a comment - To be compliant with Apache license we have to remove all references to LZO, however you can still use LZO and achieve the same behavior by adding the lines below to your hadoop-site.xml or chukwa-demux-conf.xml. <property> <name>mapred.output.compress</name> <value>true</value> <description>Should the job outputs be compressed? </description> </property> <property> <name>mapred.output.compression.type</name> <value>BLOCK</value> <description>If the job outputs are to compressed as SequenceFiles, how should they be compressed? Should be one of NONE, RECORD or BLOCK. </description> </property> <property> <name>mapred.output.compression.codec</name> <value>org.apache.hadoop.io.compress.LzoCodec</value> <description>If the job outputs are compressed, how should they be compressed? </description> </property> <property> <name>mapred.compress.map.output</name> <value>true</value> <description>Should the outputs of the maps be compressed before being sent across the network. Uses SequenceFile compression. </description> </property> <property> <name>mapred.map.output.compression.codec</name> <value>org.apache.hadoop.io.compress.LzoCodec</value> <description>If the map outputs are compressed, how should they be compressed? </description> </property>
          Hide
          eyang Eric Yang added a comment -

          +1 Looks good.

          Show
          eyang Eric Yang added a comment - +1 Looks good.
          Hide
          eyang Eric Yang added a comment -

          I just committed this, thanks Jerome.

          Show
          eyang Eric Yang added a comment - I just committed this, thanks Jerome.

            People

            • Assignee:
              jboulon Jerome Boulon
              Reporter:
              eyang Eric Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development