[FLINK-14551] Unaligned checkpoints

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.11.0
• Component/s: Runtime / Checkpointing, Runtime / Network
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Zhijiang
• Created: 29/Oct/19 07:45
• Updated: 13/Oct/20 08:23
• Resolved: 13/Oct/20 08:23

Description

This is the umbrella issue for the feature of unaligned checkpoints. Refer to the FLIP-76 for more details.

 

 

Issue Links

• is depended upon by: FLINK-19442 Unaligned checkpoints 1.1
• relates to: FLINK-17122 Support InputSelectable and BoundedMultiInput operators with checkpointing
• links to: GitHub Pull Request #11354

Sub-Tasks

• Remove unnecessary interface method BufferProvider#requestBufferBlocking: Closed
• Implement rudimentary non-blocking network output: Closed
• Introduce NetworkBufferPool#isAvailable() for non-blocking output: Closed
• Implement back-pressure monitor with non-blocking outputs: Closed
• Respect non-blocking output in StreamTask#processInput: Closed
• Add API to persist channel state: Closed
• Implement API to persist channel state: checkpointing metadata: Closed
• Implement InputChannel state recovery for unaligned checkpoint: Closed
• Implement ResultPartition state recovery for unaligned checkpoint: Closed
• Build ResultSubpartitionInfo and InputChannelInfo in respective constructors: Closed
• Add basic CheckpointBarrierHandler for unaligned checkpoint: Closed
• Implement API to persist channel state: Closed
• Add recovery to integration tests of unaligned checkpoints: Closed
• Persist spilled records of SpillingAdaptiveSpanningRecordDeserializer: Closed
• Enable unaligned checkpoints in tests by default: Closed
• Add checkpoint cancellation to Unaligner: Closed
• Refactor the usages of ThreadSafeUnaligner#storeNewBuffers: Closed
• Schedule savepoint if max-inflight-checkpoints limit is reached instead of forcing (in UC mode): Closed
• Implement aligned savepoint in UC mode: Closed
• Provide ChannelStateReader#hasStates for hints of reading channel states: Closed
• Support unaligned checkpoints for records spilled to files: Closed
• Add check for max concurrent checkpoints under UC mode: Closed
• Update documentation taking unaligned checkpoints into the account: Closed