After changed the job-dir permission to 777. Tried the following:
- Start NN, DN as nn_account (superuser by definition)
- Start JT, TT as jt_account (non-superuser)
- Run wordcount as user_account (non-superuser)
Below is an observation of the files/dir creation and their permission setting.
Step 0: During JobTracker startup
mapred-sys-dir (delete-mkdirs-setPerm as jt_account)
Step 1: Client side job submission (JobClient .submitJob)
job-dir (aka submitJobDir or mapred-sys-dir/jobId, mkdirs-setPerm as user_account)
(was 733) 777
job-dir/job.jar (may not exists, create-setPerm by JobClient)
job-dir/job.split (create-setPerm by JobClient)
job-dir/job.xml (aka jobconf, create-setPerm by JobClient; Should it be visible in JobTracker webpage??? It is visible now.)
Miscellaneous dirs: Current using the default permission, i.e. umask. What is the correct permission for them???
- filesDir = new Path(submitJobDir, "files");
- archivesDir = new Path(submitJobDir, "archives");
- libjarsDir = new Path(submitJobDir, "libjars");
Step 2: JobTracker side job submission (JobTracker.submitJob)
job-output-dir (aka mapred.output.dir)
What are the correct permissions???
They may be created by anto-mkdir in JobTracker as user_account
- For example the first file created under job-output-dir in wordcount is job-output-dir/_logs/history/hostname_1207356629390_job_200804041750_0001_username_wordcount. Note that job-output-dir/_logs/history is the job-history-dir.
- This file is created as user_account
Task does mkdirs job-output-dir as user_account
- In Task.moveTaskOutputs(...), is finalOutputPath a constant? If yes, it should not be computed again and again in Task.moveTaskOutputs(...).
job-temp-dir (aka _temporary, created in JobInProgress.<init> as user_account)
What is the correct permission???
Still need to look at the directories clean-up and the case in LocalJobRunner.