Index: downloads.html =================================================================== --- downloads.html (revision 1226503) +++ downloads.html (working copy) @@ -2,6 +2,18 @@

Downloads

-We have not yet done an Apache release and Apache does not allow us to host non-Apache releases on this site. You can download the previous releases here. +

0.7.0-incubating

+The current stable version is 0.7.0-incubating. See the Release Notes +Be sure to verify your downloads by these procedures using these KEYS.
+
+ +Download Source: kafka-0.7.0-incubating-src.tar.gz (asc, md5) + +

Previous non-incubator releases

+You can download the previous releases here. + +

DISCLAIMER

+Apache Kafka is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF. + Index: quickstart.html =================================================================== --- quickstart.html (revision 1226503) +++ quickstart.html (working copy) @@ -37,14 +37,14 @@ A toy producer script is available to send plain text messages. To use it, run the following command: 
-> bin/kafka-producer-shell.sh --server kafka://localhost:9092 --topic test
+> bin/kafka-producer-shell.sh --props config/producer.properties --topic test
 > hello
 sent: hello (14 bytes)
 > world
 sent: world (14 bytes)
-

Step 5: Start a consumer

+

Step 4: Start a consumer

Start a toy consumer to dump out the messages you sent to the console: @@ -58,7 +58,7 @@ If you have each of the above commands running in a different terminal then you should now be able to type messages into the producer terminal and see them appear in the consumer terminal. -

Step 6: Write some code

+

Step 5: Write some code

Below is some very simple examples of using Kafka for sending messages, more complete examples can be found in the Kafka source code in the examples/ directory. @@ -100,7 +100,7 @@
2. Producer API
-With release 0.6, we introduced a new producer API - kafka.producer.Producer<T>. Here are examples of using the producer - +Here are examples of using the producer API - kafka.producer.Producer<T> -
  1. First, start a local instance of the zookeeper server @@ -209,12 +209,13 @@ producer.send(data);
    2. -
  2. Use the asynchronous producer. This buffers writes in memory until either batch.size or queue.time is reached. After that, data is sent to the Kafka brokers +
  3. Use the asynchronous producer along with GZIP compression. This buffers writes in memory until either batch.size or queue.time is reached. After that, data is sent to the Kafka brokers 
     Properties props = new Properties();
 props.put("zk.connect"‚ "127.0.0.1:2181");
 props.put("serializer.class", "kafka.serializer.StringEncoder");
 props.put("producer.type", "async");
+props.put("compression.codec", "1");
 ProducerConfig config = new ProducerConfig(props);
 Producer<String, String> producer = new Producer<String, String>(config);
 ProducerData<String, String> data = new ProducerData<String, String>("test-topic", "test-message");
@@ -295,10 +296,10 @@
 
   // get the message set from the consumer and print them out
   ByteBufferMessageSet messages = consumer.fetch(fetchRequest);
-  for(Message message : messages) {
-    System.out.println("consumed: " + Utils.toString(message.payload(), "UTF-8"));
+  for(MessageAndOffset msg : messages) {
+    System.out.println("consumed: " + Utils.toString(msg.message.payload(), "UTF-8"));
     // advance the offset after consuming each message
-    offset += MessageSet.entrySize(message);
+    offset = msg.offset;
   }
 }
    Index: includes/header.html =================================================================== --- includes/header.html (revision 1226503) +++ includes/header.html (working copy) @@ -26,7 +26,7 @@
    • download
      • -
    • api docs
      • +
    • api docs
    • quickstart
    • design
    • configuration
      • Index: configuration.html =================================================================== --- configuration.html (revision 1226503) +++ configuration.html (working copy) @@ -18,6 +18,11 @@ Each broker is uniquely identified by an id. This id serves as the brokers "name", and allows the broker to be moved to a different host/port without confusing consumers. + enable.zookeeper + true + enable zookeeper registration in the server + + log.flush.interval 500 Controls the number of messages accumulated in each topic (partition) before the data is flushed to disk and made available to consumers. @@ -49,6 +54,11 @@ Topic-specific retention time that overrides log.retention.hours, e.g., topic1:10,topic2:20 + log.retention.size + -1 + the maximum size of the log before deleting it. This controls how large a log is allowed to grow + + log.cleanup.interval.mins 10 Controls how often the log cleaner checks logs eligible for deletion. A log file is eligible for deletion if it hasn't been modified for log.retention.hours hours. @@ -64,22 +74,42 @@ Controls the maximum size of a single log file. + max.socket.request.bytes + 104857600 + the maximum number of bytes in a socket request + + + monitoring.period.secs + 600 + the interval in which to measure performance statistics + + num.threads Runtime.getRuntime().availableProcessors Controls the number of worker threads in the broker to serve requests. - num.partitions + num.partitions 1 Specifies the default number of partitions per topic. + socket.send.buffer + 102400 + the SO_SNDBUFF buffer of the socket sever sockets + + + socket.receive.buffer + 102400 + the SO_RCVBUFF buffer of the socket sever sockets + + topic.partition.count.map none Override parameter to control the number of partitions for selected topics. E.g., topic1:10,topic2:20 - zk.connect + zk.connect localhost:2182/kafka Specifies the zookeeper connection string in the form hostname:port/chroot. Here the chroot is a base directory which is prepended to all path operations (this effectively namespaces all kafka znodes to allow sharing with other applications on the same zookeeper cluster) @@ -168,6 +198,26 @@ -1 By default, this value is -1 and a consumer blocks indefinitely if no new message is available for consumption. By setting the value to a positive integer, a timeout exception is thrown to the consumer if no message is available for consumption after the specified timeout value. + + rebalance.retries.max + 4 + max number of retries during rebalance + + + mirror.topics.whitelist + "" + Whitelist of topics for this mirror's embedded consumer to consume. At most one of whitelist/blacklist may be specified. + + + mirror.topics.blacklist + "" + Topics to skip mirroring. At most one of whitelist/blacklist may be specified + + + mirror.consumer.numthreads + 4 + The number of threads to be used per topic for the mirroring consumer, by default + @@ -270,7 +320,23 @@ max.message.size 1000000 the maximum number of bytes that the kafka.producer.SyncProducer can send as a single message payload - + + + compression.codec + 0 (No compression) + This parameter allows you to specify the compression codec for all data generated by this producer. + + + compressed.topics + null + This parameter allows you to set whether compression should be turned on for particular topics. If the compression codec is anything other than NoCompressionCodec, enable compression only for specified topics if any. If the list of compressed topics is empty, then enable the specified compression codec for all topics. If the compression codec is NoCompressionCodec, compression is disabled for all topics. + + + zk.read.num.retries + 3 + The producer using the zookeeper software load balancer maintains a ZK cache that gets updated by the zookeeper watcher listeners. During some events like a broker bounce, the producer ZK cache can get into an inconsistent state, for a small time period. In this time period, it could end up picking a broker partition that is unavailable. When this happens, the ZK cache needs to be updated. This parameter specifies the number of times the producer attempts to refresh this ZK cache. + +