[REEF-1223] IMRU Fault Tolerance - restart failed evaluators

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16
• Component/s: IMRU, REEF.NET
• Labels: FT
• Assignee: Julia Wang
• Reporter: Julia Wang
• Created: 28/Feb/16 00:49
• Updated: 09/Jun/17 02:49
• Resolved: 09/Jun/17 02:49

Description

Currently in .Net Group Communication and IMRU scenario, if one of the Evaluator failed for whatever reason, all the Evaluators will be killed by the driver.

There are multiple levels of fault tolerant. The scenario we would like to support in this JIRA is:

• When an evaluator failed, this failed evaluator will be killed and other good Evaluators will stay, but all the tasks running on those Evaluators will be stopped.
• A new Evaluator will be requested and started with the original task.
• Same tasks will be resubmitted to the rest the Evaluators
• The topology of those tasks will be kept in the same group communication as before
• The data that have been downloaded in those good Evaluators will stay.

Issue Links

• contains: REEF-1325 Fix PoisonTest.TestPoisonedEvaluatorStartHanlder failure in AppVeyor
• contains: REEF-1305 Moving the communication group creation before submitting tasks and decouple evaluator/context requests from task creation
• contains: REEF-1399 Node stuck in group communication failure case
• contains: REEF-1420 Dispose IGroupCommClient/Network Service from IMRU tasks
• contains: REEF-1492 On IMRU recovery: if ResultHandler.Dispose() throws exception, IMRU Driver hangs.
• contains: REEF-1677 IMRU: Evaluators failed during WaitingForEvaluator phase don't count towards MaximumNumberOfEvaluatorFailures limit
• contains: REEF-1680 Increasing default retry count in WaitingForRegistration
• contains: REEF-1685 Complete the Job properly if update/master task is completed from running state
• contains: REEF-1683 Use default MaxRetryNumberInRecovery properly
• contains: REEF-1224 IMRU Fault Tolerance - Separate Data downloading from Task injection
• contains: REEF-1511 timeout for Task Shutdown during IMRU recovery
• contains: REEF-1549 Resolve the issue in WaitingForRegistration
• contains: REEF-1550 Clean up task exceptions in IMRU task hosts
• contains: REEF-1556 Add number of forced failures for IMRU fault tolerant testing
• contains: REEF-1452 Make PoisonException Serializable
• contains: REEF-1418 IMRU State management
• contains: REEF-1304 Create tests which use .NET Poison to validate evaluator failure scenarios
• contains: REEF-1316 Adding test for resubmitting Evaluator
• contains: REEF-1317 Adding test for resubmitting tasks
• contains: REEF-1451 IMRU Fault Tolerant scenario testing
• contains: REEF-1682 Update TCP Connection config values for IMRU example and test
• contains: REEF-1366 Create tests which use .NET Poison to validate task failure scenarios
• contains: REEF-1225 IMRU Fault Tolerance - Identify the failure cases in the Group Communication and events received
• contains: REEF-1226 Evaluator Fault Tolerant - Prototype
• contains: REEF-1248 Identify the scenarios that need to restart evaluators
• contains: REEF-1249 Add REEF Poison to REEF.NET
• contains: REEF-1251 IMRU Driver handlers for Fault Tolerant
• contains: REEF-1260 Adding a sample for Context Start handler
• contains: REEF-1318 TaskSubmitor - task preparation and submission in IMRU
• contains: REEF-1320 Creating default communication group in passive way
• contains: REEF-1321 Task Manager for Fault Tolerant
• contains: REEF-1322 Allow Communication Group to be removed from IGroupCommDriver
• contains: REEF-1327 Creating task states and state transitions for the IMRU Driver
• contains: REEF-1335 Create State Machine for IMRU fault tolerance
• contains: REEF-1339 Adding IInputPartition.Cache() for data download and cache
• contains: REEF-1340 Creating Context manager for Fault Tolerant
• contains: REEF-1345 Define task exceptions for IMRU Task
• contains: REEF-1378 Evaluator Manager for IMRU
• contains: REEF-1381 Allow to add Observer for ActiveContextManager
• contains: REEF-1386 Adding ICloseEvent handler for IMRU task
• contains: REEF-1392 Adding IObserver<ICloseEvent> for IMRU tasks
• contains: REEF-1404 IMRU task state Maintenance and Preservation in Evaluator for fault tolerant
• contains: REEF-1405 Cache the data in IMRU Context layer
• contains: REEF-1408 Creat IMRU functional test infrastructure and add tests for IMRU Task close handler
• contains: REEF-1466 Cancel the blocking message reading and close the task properly
• contains: REEF-1346 Throw proper exceptions in Evaluator and Context for fault tolerant
• is blocked by: REEF-1423 Tasks are not disposed after they are closed
• is blocked by: REEF-217 Evaluator reads the driver configuration (it shouldn't)
• is blocked by: REEF-1278 IFailedEvaluator message leads to shutting down of JAVA side driver
• is blocked by: REEF-1279 Injecting RuntimeClock in event handler creates second instance of clock
• is blocked by: REEF-1388 Fix RunningTask to be sent for short-lived .NET tasks
• is blocked by: REEF-1421 Transport Client inner thread is not canceled when object is disposed
• is blocked by: REEF-1245 Upgrade Newtonsoft.Json package version to 7.0.1
• relates to: REEF-1410 Validate Task constructor failure => FailedTask Event
• relates to: REEF-1424 Validate Task StartHandler failure => FailedTask Event
• relates to: REEF-1428 Validate Task Stop failure => FailedTask Event
• relates to: REEF-1439 Validate Exception in spun off System.Threading.Thread => FailedEvaluator Event
• relates to: REEF-1447 Validate Task close failure => FailedEvaluator Event (task has not yet finished)
• relates to: REEF-1285 Fix test issue in TestSendTaskMessage
• relates to: REEF-1407 Catching exceptions in group communication are implemented incorrectly
• relates to: REEF-1343 Fix events received in case of evaluator failure
• relates to: REEF-1425 TaskClientCodeException override user task exceptions for TaskClose
• relates to: REEF-1072 Add IDriverConnection as part of evaluator configuration
• relates to: REEF-1691 Should not request extra evaluators if evaluator failed at WatingForEvaluator state
• relates to: REEF-1280 The message returned from failed evaluator doesn't contain the real exception message
• relates to: REEF-1294 Consider race condition in Evaluator.SetRuntimeHandlers
• relates to: REEF-1692 Revert the ignorance for extra Evaluators requested
• relates to: REEF-1397 Revise FileSystemInputPartition and RandomInputPartition with DataCache and DataMover
• relates to: REEF-1208 Validate that the REEF.NET Evaluator logs and reports exceptions correctly
• relates to: REEF-1267 Implement IActiveContext.SubmitContextAndService
• relates to: REEF-1268 Complete ServiceConfiguration in REEF.NET
• relates to: REEF-1365 Define semantics of IFileDeSerializer, CopyToLocal, and their usage in FileSystemInputPartition
• relates to: REEF-1357 Allow different caching levels for caching in IInputPartition
• relates to: REEF-769 Implement IFailedEvaluator
• relates to: REEF-1286 Forward .NET Exceptions from the Evaluator to the Driver
• relates to: REEF-1364 C# Evaluator should attempt to send a failure message back to the Driver on an unhandled Exception
• relates to: REEF-796 Do Avro (de)serialization for FailedTask in bridge
• relates to: REEF-1312 Convert IMRU.Examples to test
• relates to: REEF-1256 Implementing Bridge for Task close for .Net
• relates to: REEF-1257 Add TaskCloseEvent handler in TaskRuntime
• relates to: REEF-1258 Populate Task Exception data properly
• relates to: REEF-1289 Add an EvaluatorConfiguration for the .NET Evaluator
• relates to: REEF-1392 Adding IObserver<ICloseEvent> for IMRU tasks
• relates to: REEF-1238 Remove obsolete .NET code in O.A.R.Wake and O.A.R.Driver
• relates to: REEF-1565 Make Context accessible from task