[REEF-250] Support global and local Configurations

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 0.10
• Component/s: REEF
• Assignee: Unassigned
• Reporter: Markus Weimer
• Created: 15/Apr/15 20:33
• Updated: 25/Feb/17 01:14

Description

We frequently have the need to mirror parameters from the Driver Configuration into all Evaluator configurations. Examples include the job name and network parameters (REEF-246 and REEF-247).

The current approach to do so is very cumbersome. We have the parameters injected in o.a.r.runtime.common.driver.evaluator.EvaluatorManager and then bind them to the Evaluator Configuration in o.a.r..runtime.common.driver.evaluator.AllocatedEvaluatorImpl#launch. This is error-prone and tedious.

Instead, we could follow the approach we've taken for files: We support the notion of global and local files: Global files are made available on all containers (Driver and Evaluator) in the folder reef/global/ in the working directory of the container. Local files are made available only on that specific container in the folder reef/local.

We should have global and local Configuration as well, each of which is available separately for JVM and CLR containers. We can standardize the file names for them in o.a.r.runtime.common.files.REEFFileNames to make sure we don't mistype them.

What do you think?

Issue Links

• relates to: REEF-217 Evaluator reads the driver configuration (it shouldn't)