[FLINK-3347] TaskManager (or its ActorSystem) need to restart in case they notice quarantine

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.10.1
• Fix Version/s: 1.0.0, 1.1.4, 1.2.1, 1.3.0
• Component/s: Runtime / Coordination
• Assignee: Till Rohrmann
• Reporter: Stephan Ewen
• Created: 05/Feb/16 15:03
• Updated: 26/Feb/18 15:42
• Resolved: 28/Apr/17 09:53

Description

There are cases where Akka quarantines remote actor systems. In that case, no further communication is possible with that actor system unless one of the two actor systems is restarted.

The result is that a TaskManager is up and available, but cannot register at the JobManager (Akka refuses connection because of the quarantined state), making the TaskManager a useless process.

I suggest to let the TaskManager restart itself once it notices that either it quarantined the JobManager, or the JobManager quarantined it.

It is possible to recognize that by listening to certain events in the actor system event stream: http://stackoverflow.com/questions/32471088/akka-cluster-detecting-quarantined-state

Issue Links

• relates to: FLINK-8788 Enable quarantine monitor by default
• links to: GitHub Pull Request #2696
• links to: GitHub Pull Request #2697
• links to: GitHub Pull Request #3363
• links to: GitHub Pull Request #4478