[REEF-1271] Add better test reporting APIs to REEF.NET

• Type: Improvement
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Component/s: REEF.NET, Tests
• Assignee: Markus Weimer
• Reporter: Markus Weimer
• Created: 23/Mar/16 15:55
• Updated: 06/Nov/17 18:52

Description

REEF.NET currently relies on parsing Driver and Evaluator logs to determine whether a functional (integration) test passed. The test puts a specific message into the log, which is parsed after the Driver finishes.

This works OK for tests on the local runtime, but is undesirable for tests on clusters. On the Java side, we rely on REEF's forwarding of exceptions from the Evaluator to the Driver and the Driver to the Client: A test indicates failure by throwing DriverSideFailure or EvaluatorSideFailure exceptions.

Issue Links

• contains: REEF-1352 Add exception classes for integration tests
• incorporates: REEF-1948 Add IREEFClient.NewJobRequestBuilder
• is blocked by: REEF-1942 Add ability to wait for Driver to complete to REEF.NET Client
• is related to: REEF-1951 Make DriverStatus more robust
• is related to: REEF-1265 Clean up tests inherited from ReefFunctionalTests
• relates to: REEF-1352 Add exception classes for integration tests
• relates to: REEF-1449 Speed up .NET tests
• relates to: REEF-1253 Add E2E testing infrastructure

Sub-Tasks

• Create basic testing API & implementation for the local runtime: Resolved
• Implement YarnTestRunner: In Progress
• Add richer Assert APIs: Open
• Only write the assert files in DriverStop: Open