Uploaded image for project: 'REEF'
  1. REEF
  2. REEF-1787

Unmanaged REEF on Spark packages global.jar resources incorrectly

    XMLWordPrintableJSON

    Details

      Description

      When REEF application is being launched in Unmanaged AM mode from Spark driver on YARN, it packages the global resources incorrectly. As a result, REEF Evaluators cannot find REEFLauncher class and fail.

      All global resources required by REEF application are being packaged into the global.jar file on HDFS; however, when packaged from Spark, the resources include full path to the files, e.g.

      $ jar tvf global.jar
       15082 Wed Apr 19 19:03:50 PDT 2017 mnt/data/0/local/nm/usercache/hadoop/filecache/18/reef-driver-on-spark_2.11-1.01.jar
      16963704 Wed Apr 19 19:03:50 PDT 2017 mnt/data/0/local/nm/usercache/hadoop/filecache/15/reef-examples-0.16.0-SNAPSHOT-shaded.jar
      

      whereas the same global.jar produced by REEF client does not include such paths.

      We have to investigate whether absence or presence of such paths in the global.jar causes the Evaluator start failure.

        Attachments

          Activity

            People

            • Assignee:
              motus Sergiy Matusevych
              Reporter:
              motus Sergiy Matusevych
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 168h
                168h
                Remaining:
                Remaining Estimate - 168h
                168h
                Logged:
                Time Spent - Not Specified
                Not Specified