If the jar is built on a Red Hat based OS, the additional python files included in the jar cannot be accessed. This means PySpark doesn't work on YARN because in this mode it relies on the python files within this jar.
I have confirmed that my Java, Scala, and maven versions are all exactly the same on my CentOS environment and on my local OSX environment, and the former does not work. Thomas Graves also struggled with the same problem.
Until a fix is found, we should at the very least document this peculiarity.