Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Blocker
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 0.21.0
-
Component/s: tasktracker
-
Labels:None
-
Hadoop Flags:Reviewed
-
Release Note:Fixed a bug that failed jobs that are run by the same user who started the mapreduce system(cluster).
Description
Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
Directories are created as -:
[dr-xrws--- mapred hadoop ] job_200908190916_0002
mapred is not wrtie under this. Even manually I failed to touch file.
mapred is use of started mr cluster
Issue Links
- is blocked by
-
MAPREDUCE-856
Localized files from DistributedCache should have right access-control
-
- Closed
-
-
MAPREDUCE-1322
TestStreamingAsDifferentUser fails on trunk
-
- Closed
-
-
MAPREDUCE-1421
LinuxTaskController tests failing on trunk after the commit of MAPREDUCE-1385
-
- Closed
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
In other words, the problem is the following:
$ mkdir testing $ ls -ld testing/ drwxr-xr-x 2 vinodkv vinodkv 4096 2009-08-21 09:54 testing/ $ touch testing/t1.txt $ echo $? 0 $ chmod 0570 testing/ $ ls -ld testing/ dr-xrwx--- 2 vinodkv vinodkv 4096 2009-08-21 09:54 testing/ $ touch testing/t2.txt touch: cannot touch `testing/t2.txt': Permission denied
I searched and hunted this down in the Linux Kernel filesystem code: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=fs/namei.c;h=f3c5b278895a0d3e0f23fe6fd474e2728a1c6cb6;hb=6c30c53fd5ae6a99a23ad78e90c428d2c8ffb07f . Particularly the generic_permission() function where we check if current_fsuid() == inode->i_uid.
Attaching patch which should be applied over the latest patch at MAPREDUCE-856. It follows a slight modification of the second proposal above, which is
Set drwxrws--- on $job_cache and $dist_cache in all cases. This means user's tasks CAN potentially create unwarranted files/dirs in the $job_cache or $dist-cache or delete their own files themselves.
So now, the user-directory taskTracker/$user will have "2570 user-owner task-tracker-group" and all the world under it will have "2570/0770 user-owner task-tracker-group".
Updated patch to be applied over the latest patch at MAPREDUCE-856.
I think this is a blocker - when running with LinuxTaskController, it prevents job-submission as the user running the mapred daemons.
The same (latest) patch works post feature-freeze even with all the patches that went in.
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12419014/MAPREDUCE-890-20090909.txt
against trunk revision 834284.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/console
This message is automatically generated.
The patch above that implements the 2nd proposal needs more changes.
Set drwxrws--- on $job-dir in all cases. This means user's tasks CAN potentially create unwarranted files/dirs in the $job_dir
With this, unwarranted files/dirs with improper permissions in job-dir will prevent TT from recursively deleting all the files - a manifestation of MAPREDUCE-896 in the job-dir. This should also be fixed in this patch itself.
Setting drwxrws--- for jobcache and distcache directories allows users to create files in these directories and as these directories won't be cleaned up for each job(will be cleaned up only when TT is restarted), this can cause space issues.
So I propose considering tt_user running MR job as a special case and handle that in a special way. i.e. set drwxrwx--- for distcache and jobcache only for jobs of tt_user. This keeps the security of these dirs for all other users as it is there in current trunk.
Thoughts ?
Another solution is to call DefaultTaskController's methods from LinuxTaskContrller's methods if user(i.e. jobOwner) is tt_user. This will keep 700 permissions for userDir, jobcache, distcache dirs and so are secure and are accessible to jobOwner.
As I don't see any issues with either of these 2 solutions, currently I am planning to implement 1st approach(i.e. LinuxTaskController setting 770 permissions for jobcache and distcache dirs if jobOwner is tt_user).
Thoughts ?
I would like to give it some more thought. But for now, I think Approach 1 seems to be the right way to go (in spite of the special check). I am assuming the special check is in the task-controller ?
Yes Hemanth. Check is in task-controller.
Attaching patch fixing the issue.
Patch does the following:
(1) When localizing userDir, in intialize_user(), we do "chmod 2770 -R taskTracker/$user" for tt_user(i.e. if jobOwner is same as tt_user) and "chmod 2570 -R taskTracker/$user" otherwise.
(2) Similarly,
(a) "chmod 2770 -R taskTracker/$user/jobcache/$jobid" for tt_user and "chmod 2570 -R taskTracker/$user/jobcache/$jobid" otherwise.
(b) "chmod 2770 -R taskTracker/$user/distcache/<randomdir>" for tt_user and "chmod 2570 -R taskTracker/$user/distcache/<randomdir>" otherwise.
(3) compare_ownership() check was done in secure_path() sothat setting of permissions for distributed-cache files(for which permissions are already set earlier) will not go through chown and chmod again and again. This check is not needed after MAPREDUCE-1186(which is already committed to trunk). So deleted this check from secure_path().
(4) Modified TestLocalizationWithLinuxTaskController sothat validation of permissions of different files/directories will work properly in the cases of (a) jobOwner is tt_user and (b) any other user.
Please review and provide your comments.
Looked at the patch. Got some comments:
- After patch, at +643, in initialize_user() function, the patch incorrectly passes 0 to secure_path()'s shouldCheckOwnership argument.
- TestTrackerDistributedCacheManager* tests will need modifications too, I guess.
- It's already a pain running the task-controller tests. With this patch, the expectation is to run these 5-odd tests with $tt_user and $$non_tt_user, that is two times. I think we should modify the tests themselves so that this double test-run is automatically done.
As MAPREDUCE-1429 will take care of running the linux task controller based tests with both different user and tt_user(and also changing the source code of all tests is not maintainable — as future linux task controller based testcases should also consciously follow this duplication of testcases for both users), I will drop this(Vinod's 3rd comment) from this patch. Incorporating first 2 comments.
why don't we run LinuxTaskController tests as the test-launcher(thereby as TT), if nothing is passed in taskcontroller-ugi ?
I don't see much value in changing all LinuxTaskController based tests to take default value for taskcontroller-ugi when taskcontroller-path is specified(anyway user has to specify taskcontroller-path for the tests to be run) and taskcontroller-ugi is not specified.
Do you really think it is useful ?
Attaching patch with Vinod's 1st 2 review comments incorporated. I don't see value in making source code changes for 3rd comment and discussed it with Vinod offline and MAPREDUCE-1429 will take care of it, if needed.
Please review and provide your comments.
The patch is not applying over the latest patch at MAPREDUCE-1421 which is a blocker for this JIRA issue. Can you please upload an updated patch?
Syncing the patch with the current trunk.
Patch looks good to me. +1, pending MAPREDUCE-1421.
MAPREDUCE-1421 is committed. Running this patch through Hudson.
Ravi, can you run tests and test-patch? Hudson is not picking patches up, so..
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12437589/MR890.v1.1.patch
against trunk revision 918864.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 24 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/console
This message is automatically generated.
The tests TestSymLink.testSymLink, TestUlimit, TestIO failed because of the test run by Hudson could not find some of the core class. Wierd. Anyone knows why?
Filed MAPREDUCE-1562 for the failure of TestBadRecords.testBadMapRed.
Syncing with the latest trunk(as MAPREDUCE-1435 went in).
Allowing it to go through Hudson
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12437984/MR890.v1.2.patch
against trunk revision 919335.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 24 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/console
This message is automatically generated.
For some reason, Hudson's couldn't report which tests failed on its web portal, even though the console output is still there. I've downloaded the console output and saw that all the tests passed except TestTaskTrackerMemoryManager. And this failure is because of the same wierd java.lang.NoClassDefFoundError on some core classes that's been plaguing other test runs, this time on org.apache.hadoop.http.HttpServer$QuotingInputFilter.
I could run TestTaskTrackerMemoryManager successfully. I am going to check this in.
I've just committed this to trunk. Thanks Ravi!
The trunk patch isn't applying to 0.21 branch. I think we'll need a new one, can you work on it?
Attaching patch for earlier version of hadoop(includes the fix of MAPREDUCE-1573 also). Not for commit here.
Since this was already committed to trunk, I'm going to mark it as resolved purely for accounting purposes. I've also moved the entry from 0.21 to 0.22 in CHANGES.txt. Once the patch for 0.21 is committed, the entry can be moved back to 0.21.
Integrated in Hadoop-Mapreduce-trunk-Commit #285 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/285/)
Integrated in Hadoop-Mapreduce-trunk #264 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/264/)
Move from 0.21 to 0.22 in changelog to match commit
As part of
MAPREDUCE-842, job directories are set to have the following permissions:CASE I: the $job-owner is other than the $tt_user
$tt_user is part of $tt_group, and so can create attempt directories inside the job directory as part of the task-localization because job_directory is group writable.
CASE II: the $job-owner is same as the $tt_user
TT cannot create attempt directories inside the job directory!! Because Linux seems to check uid of the process with the fsuid of the directory and return error if the directory is not user writable!
CASE II is what is causing the current bug.
Alternative solutions at hand:
Thoughts?