[REEF-751] Add a standard way to provide cluster-local configuration

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: REEF Client, REEF.NET Client
• Labels: C# GSoC gsoc2016 gsoc2017 java
• Assignee: Unassigned
• Reporter: Markus Weimer
• Created: 15/Sep/15 20:28
• Updated: 12/Mar/17 16:55

Description

When running a REEF application on a specific cluster, we sometimes have to make cluster-specific configuration choices beyond what's available via e.g. the YARN Configuration. One example are clusters which only allow us to open TCP ports in a specific range.

Right now, we require the application to add such configuration to the Driver, e.g. via the DriverConfigurationProviders mechanism. This is awkward, because it limits the portability of applications: In order for an app to run such a cluster, one would have to change the app's code. This is undesirable.

Instead, we should have a standard mechanism by which one can provide additional runtime-level configuration to be picked up by the REEF client. Let's use this JIRA to discuss potential solutions to this issue.