[REEF-1274] Add validation to YarnConfigurationConstructor

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: REEF-Runtime-YARN
• Assignee: Unassigned
• Reporter: Markus Weimer
• Created: 23/Mar/16 16:38
• Updated: 24/Mar/16 17:25

Description

A frequent problem REEF applications encounter is a broken or unavailable YARN Configuration of the cluster. REEF can help with the diagnose of such issues as we always load / instantiate the YARN Configuration via o..r.runtime.yarn.util.YarnConfigurationConstructor. In that class, we can validate the YARN configuration and report WARNINGs into the log if something is odd. For example:

• Is the RM IP set and reachable?
• Is the classpath set and non-empty? If not, are the environment variables set?
• Can a `FileSystem` instance be created using that configuration?

There are probably more tests we can perform as we hear from our users.