Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.14.0
-
None
Description
It is not enough to know just the number of available buffers (credits) for the downstream because the size of these buffers can be different. So we are proposing to resolve this problem in the following way: If the downstream buffer size is changed then the upstream should send the buffer of the size not greater than the new one regardless of how big the current buffer on the upstream. (pollBuffer should receive parameters like bufferSize and return buffer not greater than it)
Downstream will be able to support any buffer size < max buffer size, so it should be just good enough to request BufferBuilder with new size after getting announcement, and leaving existing BufferBuilder/BufferConsumers unchanged. In other words code in PipelinedSubpartition(View) doesn’t need to be changed (apart of forwarding new buffer size to the BufferWritingResultPartition). All buffer size adjustments can be implemented exclusively in BufferWritingResultPartition.
If different downstream subtasks have different throughput and hence different desired buffer sizes, then a single upstream subtask has to support having two different subpartitions with different buffer sizes.
Attachments
Issue Links
- links to