Details
Description
According to the Spark YARN doc page, Spark executors will use the local directories configured for YARN, not spark.local.dir which should be ignored.
However it should be noted that in yarn-client mode, though the executors will indeed use the local directories configured for YARN, the driver will not, because it is not running on the YARN cluster; the driver in yarn-client will use the local directories defined in spark.local.dir
Can this please be clarified in the Spark YARN documentation above?