We've observed that netty uses large amount of onheap memory in its pools, in addition to the expected offheap memory when I added some instrumentation (using
SPARK-24918 and https://github.com/squito/spark-memory). We should figure out why its using that memory, and whether its really necessary.
which means that even with a small burst of messages, each arena will grow by 16MB which could lead to a 128 MB spike of an almost entirely unused pool. Switching to requesting a buffer from the default pool would probably fix this.