[FLINK-2929] Recovery of jobs on cluster restarts

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.0
• Fix Version/s: 1.1.0
• Assignee: Unassigned
• Reporter: Ufuk Celebi
• Created: 27/Oct/15 16:49
• Updated: 22/Jul/16 19:49
• Resolved: 22/Jul/16 19:49

Description

Recovery information is stored in ZooKeeper under a static root like /flink. In case of a cluster restart without canceling running jobs old jobs will be recovered from ZooKeeper.

This can be confusing or helpful depending on the use case.

I suspect that the confusing case will be more common.

We can change the default cluster start up (e.g. new YARN session or new ./start-cluster call) to purge all existing data in ZooKeeper and add a flag to not do this if needed.

[e-mail], aljoscha, StephanEwen what's your opinion?