[REEF-1304] Create tests which use .NET Poison to validate evaluator failure scenarios

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.15
• Component/s: REEF.NET
• Labels: FT
• Assignee: Mariia Mykhailova
• Reporter: Mariia Mykhailova
• Created: 31/Mar/16 18:19
• Updated: 17/May/16 19:40
• Resolved: 09/May/16 18:43

Description

There are multiple scenarios of failures in our system. We should use REEF.NET Poison (delayed fault injection mechanism) to simulate these scenarios in test and to validate the response of our system to them.

Here is initial list of tests we need to have.

Evaluator Failure

We expect to receive only FailedEvaluator event, no FailedContext and no FailedTask

N	Failure injected at	Additional verification
1	ON_EVALUATOR_ALLOCATED
2	between ON_EVALUATOR_ALLOCATED and ON_CONTEXT_ACTIVE
3	ON_CONTEXT_ACTIVE	FailedEvaluator should have FailedContexts filled
4	between ON_CONTEXT_ACTIVE and ON_TASK_RUNNING	FailedEvaluator should have FailedContexts filled
5	ON_TASK_RUNNING	FailedEvaluator should have FailedContexts and FailedTask filled
6	between ON_TASK_RUNNING and ON_TASK_COMPLETED	FailedEvaluator should have FailedContexts and FailedTask filled
7	ON_TASK_COMPLETED	FailedEvaluator should have FailedContexts filled
8	after ON_TASK_COMPLETED	race against evaluator completion

Issue Links

• Is contained by: REEF-1223 IMRU Fault Tolerance - restart failed evaluators
• is related to: REEF-1286 Forward .NET Exceptions from the Evaluator to the Driver
• requires: REEF-1249 Add REEF Poison to REEF.NET