When running a REEF application on a specific cluster, we sometimes have to make cluster-specific configuration choices beyond what's available via e.g. the YARN Configuration. One example are clusters which only allow us to open TCP ports in a specific range.
Right now, we require the application to add such configuration to the Driver, e.g. via the DriverConfigurationProviders mechanism. This is awkward, because it limits the portability of applications: In order for an app to run such a cluster, one would have to change the app's code. This is undesirable.
Instead, we should have a standard mechanism by which one can provide additional runtime-level configuration to be picked up by the REEF client. Let's use this JIRA to discuss potential solutions to this issue.