[OAK-6565] GetBlobResponseEncoder should not write all chunks at once

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.7.6
• Fix Version/s: 1.7.6, 1.8.0
• Component/s: segment-tar
• Labels: cold-standby
• Assignee: Andrei Dulceanu
• Reporter: Andrei Dulceanu
• Created: 18/Aug/17 13:31
• Updated: 04/Oct/19 17:49
• Resolved: 18/Aug/17 15:55

Description

GetBlobResponseEncoder writes too fast all the chunks, leaving the channel in a not-writable state, after the first write. The problem is not visible at a first glance, especially when using small blobs for testing. Increasing the blobs size, as done for OAK-6538, revealed the problem. Not only this triggers hidden OutOfMemory errors on either server or client, but sometimes incomplete blobs are sent along, which are interpreted by the client as valid.

A more elegant solution, which also solves the memory consumption problem, would be to use ChunkedWriteHandler which employs complex logic on how and when to write the chunks. ChunkedWriteHandler must be used in conjunction with a custom ChunkedInput<ByteBuf> implementation to generate header + payload chunks from an InputStream, as done currently. This way the server will send more chunks only when the previous one was consumed by the client.

/cc frm

Issue Links

• relates to: OAK-6538 Investigate cold standby memory consumption