[ACCUMULO-2360] Need a way to configure TNonblockingServer.maxReadBufferBytes to prevent OOMs from network misbehavour

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.5.0
• Fix Version/s: 1.5.1, 1.6.0
• Component/s: master, tserver
• Assignee: John Vines
• Reporter: John Vines
• Created: 12/Feb/14 19:00
• Updated: 12/Jun/14 18:18
• Resolved: 12/Feb/14 19:26

Description

1.5.0 introduced GENERAL_MAX_MESSAGE_SIZE (ACCUMULO-1141), a parameter to set the maximum frame size for the TFramedTransport. However, there is an underlying frame (I think this is a glossary conflict) read in TNonblockingServer that can still cause OOM errors if erroneously connected to (telnet, netcat, etc.), creating a stack trace as such

2014-02-12 10:26:40,439 [util.TServerUtils$THsHaServer] ERROR: run() exiting due to uncaught error
java.lang.OutOfMemoryError: Java heap space
        at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
        at java.nio.ByteBuffer.allocate(ByteBuffer.java:329)
        at org.apache.thrift.server.AbstractNonblockingServer$FrameBuffer.read(AbstractNonblockingServer.java:338)
        at org.apache.thrift.server.AbstractNonblockingServer$AbstractSelectThread.handleRead(AbstractNonblockingServer.java:202)
        at org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.select(TNonblockingServer.java:198)
        at org.apache.thrift.server.TNonblockingServer$SelectAcceptThread.run(TNonblockingServer.java:154)

I believe if we set maxReadBufferBytes to the server arguments, it will filter appropriately. The only decision I'm not sure about is if we should recycle the max message property or have a separate one.

Issue Links

• is duplicated by: ACCUMULO-2351 Master memory leak
• is related to: ACCUMULO-2897 Thrift proxy in Accumulo throws out of memory error on malformed request
• relates to: ACCUMULO-2367 Revisit thrift settings