Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-11818

ExecutorClassLoader cannot see any resources from parent class loader

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.1
    • Fix Version/s: 1.6.0
    • Component/s: Spark Shell
    • Labels:
      None
    • Environment:

      CentOS 6, spark 1.4.1-hadoop2.4, mesos 0.22.1

      Description

      This issue starts from finding root reason from strange problem from spark-shell (and zeppelin) which is not a problem for spark-submit.

      https://mail-archives.apache.org/mod_mbox/spark-user/201511.mbox/%3CCAF5108jMXyOjiGmCgr%3Ds%2BNvTMcyKWMBVM1GsrH7Pz4xUj48LfA%40mail.gmail.com%3E

      After some hours (over days) digging into the detail, I found that ExecutorClassLoader cannot see any resource files which can be seen from parent class loader.
      ExecutorClassLoader itself doesn't need to lookup resource files cause REPL doesn't generate these, but it should delegate lookup to parent class loader.

      I'll provide the pull request which includes tests which fails on master soon.

        Attachments

          Activity

            People

            • Assignee:
              kabhwan Jungtaek Lim
              Reporter:
              kabhwan Jungtaek Lim
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: