[FLINK-7315] use flink's buffers in netty

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.0
• Component/s: Runtime / Network
• Assignee: Nico Kruber
• Reporter: Nico Kruber
• Created: 31/Jul/17 14:17
• Updated: 20/Oct/20 06:57
• Resolved: 20/Oct/20 06:57

Description

The goal of this change is to avoid the step in the channel encoder and decoder pipelines where flink buffers are copied into netty buffers. Instead, netty should directly send flink buffers to the network.

Issue Links

• is duplicated by: FLINK-7512 avoid unnecessary buffer copies during network serialization

Sub-Tasks

• always use HybridMemorySegment: Closed
• activate checkstyle for flink/core/memory/*: Closed
• always use off-heap network buffers: Closed
• minor performance improvements in NettyMessage: Resolved
• optimise NettyMessage.TaskEventRequest#readFrom() to read from netty buffers directly: Resolved
• integrate PartitionRequestProtocol into NettyProtocol: Closed
• avoid one additional buffer copy when receiving messages: Closed
• double buffer release in SpillableSubpartitionView: Closed
• remove TestBufferFactory#MOCK_BUFFER: Resolved
• fix BackPressureStatsTrackerITCase releasing buffers twice: Resolved
• allow actual 0-length content in NettyMessage#allocateBuffer(): Resolved
• HybridMemorySegment: do not allow copies into a read-only ByteBuffer: Resolved
• let NettyBufferPool extend PooledByteBufAllocator: Closed
• let our Buffer class extend from netty's buffer class: Closed
• Create (derived) duplicate Buffer class: Closed
• Create (derived) sliced Buffer class: Closed
• pass our own buffer instances to netty: Closed
• add tests for ensuring NetworkBufferPool overprovisioning behaviour: Resolved
• Let Netty use Flink's buffers directly in credit-based mode: Closed