[REEF-1561] REEF as a library

• Type: New Feature
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Component/s: REEF Driver, REEF Evaluator
• Labels: refactoring
• Assignee: Sergiy Matusevych
• Reporter: Sergiy Matusevych
• Created: 12/Sep/16 19:49
• Updated: 11/Oct/16 20:10

Description

Currently REEF Driver and Evaluator start only as an application, that is, the only entry point is static main(). We need to refactor the code so we can start REEF components from another Java or C# application in the same VM.

Issue Links

• is blocked by: REEF-1637 When launched via REEFEnvironment, Driver does not receive heartbeats from Evaluators

Sub-Tasks

• Refactor logic related to Evaluator state: Resolved
• Refactor REEF Driver logic for code readability and add more javadocs: Resolved
• Refactor HandlerContainer to improve safety, readability, and performance: Resolved
• Make RuntimeClock more testable and create unit tests to check graceful and forceful shutdown of RuntimeClock: Closed
• Graceful shutdown of the RuntimeClock is not so graceful: unconditionally clears the event queue: Resolved
• Improve readability of code and logs in REEF launcher: Resolved
• Split REEFLauncher into executable part and the Clock-running environment: Resolved
• Implement proper management of all threads running in REEF and its applications: In Progress
• Get rid of global exception handlers and other process-global features in REEF: In Progress
• Add the ability to check the status of the REEF job launched directly via REEFEnvironment Driver.: Resolved
• Create more unit tests to verify error handling in the driver launched via REEFEnvironment: Resolved
• Assign name and thread group to each thread that manages local containers on the Driver side: Resolved
• Refactor RunnableProcess and move State and related logic into a separate enum: Resolved
• Check for singletons per REEFEnvironment, not per entire JVM process: Resolved
• Implement timeout for FailClient.runInProcess() and other similar unit tests: Open
• Improve logging in REEF job shutdown: Resolved
• Improve logging in REEF Remote Manager and related classes: Resolved
• Implement graceful shutdown of Wake executor services: Resolved
• Close the driver restart manager on driver shutdown: Resolved
• Implement proper logging of threads still running at the end of the REEF process: Resolved
• Give REEF threads better names and improve logging during the closing phase of REEF application: Resolved
• Gracefully shut down all threads at the end of the REEF job: Resolved
• RejectedExecutionException thrown when closing the acceptor in NettyMessageTransport: Resolved
• REEF-on-REEF example: Resolved
• Allow REEF Driver to run in unmanaged AM mode: Resolved
• Create a variant of YarnClientConfiguration for Unmanaged AM: Resolved