Batching is one of the big drivers of efficiency, and to enable batching the Kafka producer has an asynchronous mode that accumulates data in memory and sends out larger batches in a single request. The batching can be configured to accumulate no more than a fixed number of messages and to wait no longer than some fixed latency bound (say 100 messages or 5 seconds). This allows the accumulation of more bytes to send, and few larger I/O operations on the servers. Since this buffering happens in the client it obviously reduces the durability as any data buffered in memory and not yet sent will be lost in the event of a producer crash.
-Note that as of Kafka 0.8.1 the async producer does not have a callback, which could be used to register handlers to catch send errors. Adding such callback functionality is proposed for Kafka 0.9, see Proposed Producer API. +Note that kafka 0.8.2.0 releases new Java producer, which can be used to register callbacks/handlers to catch send errors. New producer functionality proposal is here Proposed Producer API.
log.cleaner.enable=trueThis will start the pool of cleaner threads. To enable log cleaning on a particular topic you can add the log-specific property
log.cleanup.policy=compactIndex: ops.html =================================================================== --- ops.html (revision 1659006) +++ ops.html (working copy) @@ -42,7 +42,8 @@
> bin/kafka-topics.sh --zookeeper zk_host:port/chroot --delete --topic my_topic_name-WARNING: Delete topic functionality is beta in 0.8.1. Please report any bugs that you encounter on the mailing list or JIRA. +Topic deletion option is disabled by default. To enable it set the server config +
delete.topic.enable=true
Kafka does not currently support reducing the number of partitions for a topic or changing the replication factor. Index: quickstart.html =================================================================== --- quickstart.html (revision 1659006) +++ quickstart.html (working copy) @@ -4,11 +4,11 @@
-> tar -xzf kafka_2.9.2-0.8.1.1.tgz -> cd kafka_2.9.2-0.8.1.1 +> tar -xzf kafka_2.10-0.8.2.0.tgz +> cd kafka_2.10-0.8.2.0