[REEF-976] Fix broken C# Tests caused by race condition of local RM

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.15
• Component/s: REEF-Tests, REEF.NET
• Labels: FT
• Assignee: Andrew Chung
• Reporter: Andrew Chung
• Created: 19/Nov/15 17:35
• Updated: 04/Apr/16 20:09
• Resolved: 04/Apr/16 20:09

Description

There is a race condition in REEF-Local-Runtime, and it can happen as follows:

1. The Evaluator sends the DONE message and exits its process.
2. The RM discovers Evaluator ends, sends DONE message to Driver.
3. Driver first gets DONE message from RM before getting reading the DONE message from the Evaluator in its network queue.
4. Driver calls FailedEvaluatorHandler, even though the Evaluator shuts down properly.

This can be fixed by requiring an ACK from the Driver prior to letting the Evaluator exit its process.

Issue Links

• is related to: REEF-1310 The Java Driver should ACK the Java Evaluator's DONE heartbeat
• relates to: REEF-347 Configure .NET tests to only listen on 127.0.0.1
• supercedes: REEF-1302 BroadcastReduceTest fail intermitently
• supercedes: REEF-977 Fix TestBroadcastAndReduceOnLocalRuntime
• supercedes: REEF-978 Fix PipelinedBroadcastReduceTest
• supercedes: REEF-1291 Driver gets FailedEvaluator message even if evaluator shuts down properly
• links to: Link to PR

Sub-Tasks

• Fix TestBroadcastAndReduceOnLocalRuntime: Closed
• Fix PipelinedBroadcastReduceTest: Closed