[FLINK-13435] Remove ShuffleDescriptor.ReleaseType and make release semantics fixed per partition type

• Type: Sub-task
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.9.0
• Fix Version/s: 1.9.0
• Component/s: Runtime / Coordination, Runtime / Network
• Labels: pull-request-available
• Assignee: Andrey Zagrebin
• Reporter: Andrey Zagrebin
• Created: 26/Jul/19 09:51
• Updated: 02/Oct/19 17:50
• Resolved: 01/Aug/19 11:50

Description

In a long term we do not need auto-release semantics for blocking (persistent) partition. We expect them always to be released externally by JM and assume they can be consumed multiple times.

The pipelined partitions have always only one consumer and one consumption attempt. Afterwards they can be always released automatically.

ShuffleDescriptor.ReleaseType was introduced to make release semantics more flexible but it is not needed in a long term.

FORCE_PARTITION_RELEASE_ON_CONSUMPTION was introduced as a safety net to be able to fallback to 1.8 behaviour without the partition tracker and JM taking care about blocking partition release. We can make this option specific for NettyShuffleEnvironment which was the only existing shuffle service before. If it is activated then the blocking partition is also auto-released on a consumption attempt as it was before. The fine-grained recovery will just not find the partition after the job restart in this case and will restart the producer.

Issue Links

• links to: GitHub Pull Request #9272