I modified YarnChild.configureLocalDirs(...) to mimic what JobLocalizer.createWorkDir(...) did in 20.205. Namely, to use the LocalDirAllocator to get a local path within one of the "mapreduce.cluster.local.dir" locations and create a scratch directory under it.
On a 10-node secure cluster, I ran manual tests to make sure that the "mapreduce.job.local.dir" was being set and that the directory was being created.
My manual tests printed the value of "job.local.dir" which showed up in the task logs. I also made sure that the scratch directories existed and were writeable by the user.
The 20.205 value for "job.local.dir" looked something like this:
The 23 value for "mapreduce.job.local.dir" looks something like this:
This solution works as far as it goes. However, I do have one concern.
In 20.205, it appears that the "job.local.dir" has the same parent dir (e.g. /cluster/0) for all of the task attempts that are run on a specific node. However, this is not the case in 23. That is, on 23, even if two task attempts are run on the same node for application_5678_0002, they could have different root directories (e.g., one task attempt would have /cluster/2 for its root dir and the other would have /cluster/4).
Is this expected behavior, or is YarnChild the wrong place to set this value?