[REEF-63] Support cross-platform job submissions to YARN clusters

• Type: New Feature
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: REEF-Runtime-YARN
• Assignee: Unassigned
• Reporter: Markus Weimer
• Created: 03/Dec/14 18:45
• Updated: 23/Sep/15 07:30

Description

When submitting an application to YARN, we need to set the command line to execute. In the case of REEF, that would be the commandline that launches the JobDriver. Several components of that command line are not universal across clusters. For example

• The location of the java binary
• The way paths are assembled (UNIX uses {:}, Windows {;}

  )

Today, the REEF YARN runtime assumes that jobs are submitted from the same platform as the one the cluster runs. And even with the same platform, there are failure cases when the location of java is different between the submitting machine and the one ultimately running the JobDriver.

With the HDInsight runtime, we could address this by hard-coding the command line to that environment. That is safe, because it doesn't matter where you are submitting from. The job will always run on a HDInsight cluster.

How do we resolve this in the general case? Shall we just add configuration options for the various choices to the YARN runtime? Is there a way to discover these things via the YARN protocol?

This used to be #913

Issue Links

• is duplicated by: REEF-322 HelloHDInsight does not support cross-platform job submission