[REEF-936] Support scale-down in REEF

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: REEF
• Assignee: Boris Shulman
• Reporter: Markus Weimer
• Created: 11/Nov/15 18:07
• Updated: 07/May/16 00:28

Description

The minimal useful REEF job right now consists of two containers: One for the Driver and one for the Evaluator. This is a fitting design for applications that need to scale out.

However, some apps might need to be elastic to workloads the size of a single Task, at which point the overhead for the Driver becomes substantial.

While there is no plan of action yet, let's use this JIRA as the link target for future work towards this.

Issue Links

• incorporates: REEF-1382 Bootstrap code mixes client and driver code

Sub-Tasks

• Add Runtime Identifier to the EvaluatorDescriptor: Resolved
• Develop status APIs for querying local containers: Open
• Write logs to Yarn logs from the local containers: Open
• Add Runtime Identifier to the EvaluatorDescriptor C#: Resolved
• Adding Project for the multi runtime: Resolved
• c# client implementation for multi-runtime: Closed
• Extensibility point for driver configuration for JobSubmissionHandlers: Resolved
• Allow specifying runtime name in evaluator request: Resolved
• multi-runtime implementation: Resolved
• Create better sample for multi-runtime: Open
• Use Configuration schema instead of string in RuntimeDefinition.avsc: Closed
• Enable specifying runtime for evaluator in the c#: Resolved
• Java yarn bootstrap for c# multi-runtime client: Resolved
• Adding extensibility to the YarnReefDotNetClient for Multi Runtime: Open
• Create MultiRuntime Configuration Builder for c#: Open
• Improve MultiRuntimeConfigurationBuilder API: Open