[REEF-1787] Unmanaged REEF on Spark packages global.jar resources incorrectly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.16
• Component/s: REEF Driver, REEF Runtime YARN
• Labels: jar resource spark yarn

Environment

YARN 2.7.3+

• Assignee: Sergiy Matusevych
• Reporter: Sergiy Matusevych
• Created: 26/Apr/17 17:51
• Updated: 02/Aug/17 02:09
• Resolved: 08/May/17 21:07

Description

When REEF application is being launched in Unmanaged AM mode from Spark driver on YARN, it packages the global resources incorrectly. As a result, REEF Evaluators cannot find REEFLauncher class and fail.

All global resources required by REEF application are being packaged into the global.jar file on HDFS; however, when packaged from Spark, the resources include full path to the files, e.g.

$ jar tvf global.jar
 15082 Wed Apr 19 19:03:50 PDT 2017 mnt/data/0/local/nm/usercache/hadoop/filecache/18/reef-driver-on-spark_2.11-1.01.jar
16963704 Wed Apr 19 19:03:50 PDT 2017 mnt/data/0/local/nm/usercache/hadoop/filecache/15/reef-examples-0.16.0-SNAPSHOT-shaded.jar

whereas the same global.jar produced by REEF client does not include such paths.

We have to investigate whether absence or presence of such paths in the global.jar causes the Evaluator start failure.