Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-634

ConsoleProducer compresses messages and ignores the --compress flag

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.7
    • Fix Version/s: 0.8.0
    • Component/s: core
    • Labels:

      Description

      I am using the kafka-producer-shell.sh script without the --compress option

      however my messages seem to be gzipped

      the docs say compression is off by default:
      http://incubator.apache.org/kafka/configuration.html

      The only producer.properties file I can find is at:
      /home/ubuntu/kafka-0.7.2-incubating-src/config/producer.properties

      In there is:
      compression.codec=0

      My process looks like:

      root 1748 1746 0 Nov19 ? 00:02:37 java -Xmx512M -server -Dlog4j.configuration=file:/usr/local/bin/kafka/../config/log4j.properties -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -cp :/usr/local/bin/kafka/../project/boot/scala-2.8.0/lib/scala-compiler.jar:/usr/local/bin/kafka/../project/boot/scala-2.8.0/lib/scala-library.jar:/usr/local/bin/kafka/../core/target/scala_2.8.0/kafka-0.7.2.jar:/usr/local/bin/kafka/../core/lib/*.jar:/usr/local/bin/kafka/../perf/target/scala_2.8.0/kafka-perf-0.7.2.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/jopt-simple-3.2.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/log4j-1.2.15.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/snappy-java-1.0.4.1.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/zkclient-0.1.jar:/usr/local/bin/kafka/../core/lib_managed/scala_2.8.0/compile/zookeeper-3.3.4.jar kafka.producer.ConsoleProducer --topic logtail --zookeeper x.x.x.x:2181

      But the messages come out gobbledegook unless I use a client that understands compressed messages, and in that client it identifies the bit as set to 1, gzip compression.

      Jun Rao junrao@gmail.com via incubator.apache.org
      Nov 26 (1 day ago)
      to kafka-users

      This seems to be a bug in ConsoleProducer. It also compresses messages and
      ignores the --compress flag. Could you file a jira?

      Thanks,
      Jun

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                anentropic Anentropic
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: