[REEF-55] Delay the creation and upload of the Global JAR in the YARN runtime Driver

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11
• Component/s: REEF-Runtime-YARN
• Labels: Performance
• Assignee: Markus Weimer
• Reporter: Markus Weimer
• Created: 02/Dec/14 22:47
• Updated: 16/Feb/15 05:07
• Resolved: 22/Jan/15 17:50

Description

EvaluatorSetupHelper is constructed as part of the YARN support in the Driver. The constructor of this class calls the .setup() which in turn (1) creates a JAR file with the global files for (future) Evaluator submissions and (2) Uploads it to the job folder on (H)DFS.

This adds significant delay in the construction of the object graph for the Driver. Hence, the code that creates and uploads the JAR file should be moved to a later stage of the computation, e.g. upon first call of getGlobalResources().

Also, it would be wise to move the code into a Callable or Runnable such that we can move it to a thread as a further optimization after this one is done.

Issue Links

• links to: This is addresses in #28