[KAFKA-6980] Recommended MaxDirectMemorySize for consumers

• Type: Wish
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 0.10.2.0
• Component/s: consumer, documentation
• Labels: consumer docuentation
• Environment: CloudFoundry
• Assignee: Unassigned
• Reporter: John Lu
• Created: 01/Jun/18 13:38
• Updated: 28/Jun/18 14:54

Description

We are observing that when MaxDirectMemorySize is set too low, our Kafka consumer threads are failing and encountering the following exception:

java.lang.OutOfMemoryError: Direct buffer memory

Is there a way to estimate how much direct memory is required for optimal performance?  In the documentation, it is suggested that the amount of memory required is  [Number of Partitions * max.partition.fetch.bytes]. 

When we pick a value slightly above that, we no longer encounter the error, but if we double or triple the number, our throughput improves drastically.  So we are wondering if there is another setting or parameter to consider?