Details
Description
Related to ACCUMULO-2360 and ACCUMULO-2352.
There are 2 thrift options in play that we use with the max message size (one introduced under 2360). The new setting is used for reading chunks of data of the network. Specifically, it gets a size, allocates a buffer, and then reads the remainder of that chunk into that buffer. If it gets garbage data that is a positive int, it will allocate that data.
It is then on top of that layer where it will reassemble frames (TFramedTransport). This is where we had the old setting. Specifically, it makes sure that really large thrift calls get rejected (with a max message size of 1g, the default, I could send a create table request up to 1g without it being outright rejected). This frame consists of smaller chunks, so the setting in the first paragraph isn't in play.
What this means is that, contrary to the logic in 2360, we should have two settings. The former, I believe, it just used to handle noise on the network without breaking things. And the size of chunks things get written in, but I'm not quite sure as I couldn't find the opposing code for the write. The latter is used for enforcing decent sized requests. So we should probably have a single value for all lowest level thrift pools (MAX_BUFFER_SIZE), but have different knobs for each service because the tserver will get the largest frames (mutations) while the master, gc, and tracer expect substantially smaller requests.
Also, I think the tracer is vulnerable to 2360 as well, since it seems to have no frame size limiting.
Attachments
Issue Links
- is related to
-
ACCUMULO-2360 Need a way to configure TNonblockingServer.maxReadBufferBytes to prevent OOMs from network misbehavour
-
- Resolved
-
-
ACCUMULO-2352 Master should have it's own property for max message size
-
- Resolved
-