I have 6 topics each with 8 partitions spread over 4 kafka servers.
the servers are 24 core 72 gig ram.
While consuming from the topics I get an IlegalArgumentException and all consumption stops, the error keeps on throwing.
I've tracked it down to FectchResponse.scala line 33
The error happens when the FetchResponsePartitionData object's readFrom method calls:
I put in some debug code the the messageSetSize is 671758648, while the buffer.capacity() gives 155733313, for some reason the buffer is smaller than the required message size.
I don't know the consumer code enough to debug this. It doesn't matter if compression is used or not.