Passing dependencies via the SPARK_SUBMIT_OPTIONS seems to be broken in 0.6.1 and 0.7.0 when using Spark 2.0. To replicate pass in a package to SPARK_SUBMIT_OPTIONS like below:
Now try to import it with:
It will error out:
<console>:25: error: object databricks is not a member of package com
I checked the logs are there is no error retrieving the package. So it seems to be something with the classpath.