Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
2.3.0
-
None
Description
When YARN runs the application's main command, it does it like this:
bash -c "<your command goes here>"
The way Spark injects the library path into that command makes it look like this:
bash -c "LD_LIBRARY_PATH="/foo:/bar:/baz:$LD_LIBRARY_PATH" <rest of the command>"
So that works kinda out of luck, because the concatenation of the strings creates a proper final command... except if you have something like a space or an ampersand in the library path, in which case all containers will fail with a cryptic message like the following:
WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Executor for container container_1475411358336_0010_01_000002 exited because of a YARN event (e.g., pre-emption) and not because of an error in the running job.
And no useful log output.
