Description
Migration tool maintains an internal blocking queue to buffer messages between the 0.7 consumer and the 0.8 producer. It is fixed in size and it set to the number of producers. However, the consumer usually fetches data in large chunks, several thousands of messages. Also each producer's internal queue frees up in chunks of the producer's batch size. In performance profiling, I saw that the migration tool's consumer thread spent time waiting for space to free up in the internal queue. It seems that this internal queue should be configurable and much larger than the number of producers. This will improve the buffering and help bridge the gap between the consumer and producer's throughput.