Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-890

After HADOOP-4491, the user who started mapred system is not able to run job.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: tasktracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed
    • Release Note:
      Fixed a bug that failed jobs that are run by the same user who started the mapreduce system(cluster).

      Description

      Even setup and cleanup task of job fails due exception -: It fails to create job and related directories under mapred.local.dir/taskTracker/jobcache
      Directories are created as -:
      [dr-xrws--- mapred hadoop ] job_200908190916_0002
      mapred is not wrtie under this. Even manually I failed to touch file.
      mapred is use of started mr cluster

      1. MAPREDUCE-890-20090904.txt
        8 kB
        Vinod Kumar Vavilapalli
      2. MAPREDUCE-890-20090909.txt
        8 kB
        Vinod Kumar Vavilapalli
      3. MR890.20S.patch
        24 kB
        Ravi Gummadi
      4. MR890.patch
        12 kB
        Ravi Gummadi
      5. MR890.v1.1.patch
        22 kB
        Ravi Gummadi
      6. MR890.v1.2.patch
        24 kB
        Ravi Gummadi
      7. MR890.v1.patch
        14 kB
        Ravi Gummadi

        Issue Links

          Activity

          Hide
          Vinod Kumar Vavilapalli added a comment -

          As part of MAPREDUCE-842, job directories are set to have the following permissions:

          permissions user ownership group ownership file/dir name
          dr-xrws--- $job-owner $tt_group job_200908190916_0002

          CASE I: the $job-owner is other than the $tt_user
          $tt_user is part of $tt_group, and so can create attempt directories inside the job directory as part of the task-localization because job_directory is group writable.

          CASE II: the $job-owner is same as the $tt_user
          TT cannot create attempt directories inside the job directory!! Because Linux seems to check uid of the process with the fsuid of the directory and return error if the directory is not user writable!

          CASE II is what is causing the current bug.

          Alternative solutions at hand:

          • Leave the code as is, and live with the fact that user's cannot submit jobs as mapred user
          • Set drwxrws--- on $job-dir in all cases. This means user's tasks CAN potentially create unwarranted files/dirs in the $job_dir
          • Set drwxrws-- if the $job-owner is same as the $tt_user, set dr-xrws--- otherwise. Handles both the cases, but complicates code very slightly.

          Thoughts?

          Show
          Vinod Kumar Vavilapalli added a comment - As part of MAPREDUCE-842 , job directories are set to have the following permissions: permissions user ownership group ownership file/dir name dr-xrws--- $job-owner $tt_group job_200908190916_0002 CASE I: the $job-owner is other than the $tt_user $tt_user is part of $tt_group, and so can create attempt directories inside the job directory as part of the task-localization because job_directory is group writable. CASE II: the $job-owner is same as the $tt_user TT cannot create attempt directories inside the job directory!! Because Linux seems to check uid of the process with the fsuid of the directory and return error if the directory is not user writable! CASE II is what is causing the current bug. Alternative solutions at hand: Leave the code as is, and live with the fact that user's cannot submit jobs as mapred user Set drwxrws--- on $job-dir in all cases. This means user's tasks CAN potentially create unwarranted files/dirs in the $job_dir Set drwxrws-- if the $job-owner is same as the $tt_user, set dr-xrws--- otherwise. Handles both the cases, but complicates code very slightly. Thoughts?
          Hide
          Vinod Kumar Vavilapalli added a comment -

          In other words, the problem is the following:

          $ mkdir testing
          $ ls -ld testing/
          drwxr-xr-x 2 vinodkv vinodkv 4096 2009-08-21 09:54 testing/
          $ touch testing/t1.txt
          $ echo $?
          0
          $ chmod 0570 testing/
          $ ls -ld testing/
          dr-xrwx--- 2 vinodkv vinodkv 4096 2009-08-21 09:54 testing/
          $ touch testing/t2.txt
          touch: cannot touch `testing/t2.txt': Permission denied
          

          I searched and hunted this down in the Linux Kernel filesystem code: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=fs/namei.c;h=f3c5b278895a0d3e0f23fe6fd474e2728a1c6cb6;hb=6c30c53fd5ae6a99a23ad78e90c428d2c8ffb07f . Particularly the generic_permission() function where we check if current_fsuid() == inode->i_uid.

          Show
          Vinod Kumar Vavilapalli added a comment - In other words, the problem is the following: $ mkdir testing $ ls -ld testing/ drwxr-xr-x 2 vinodkv vinodkv 4096 2009-08-21 09:54 testing/ $ touch testing/t1.txt $ echo $? 0 $ chmod 0570 testing/ $ ls -ld testing/ dr-xrwx--- 2 vinodkv vinodkv 4096 2009-08-21 09:54 testing/ $ touch testing/t2.txt touch: cannot touch `testing/t2.txt': Permission denied I searched and hunted this down in the Linux Kernel filesystem code: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=fs/namei.c;h=f3c5b278895a0d3e0f23fe6fd474e2728a1c6cb6;hb=6c30c53fd5ae6a99a23ad78e90c428d2c8ffb07f . Particularly the generic_permission() function where we check if current_fsuid() == inode->i_uid.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Attaching patch which should be applied over the latest patch at MAPREDUCE-856. It follows a slight modification of the second proposal above, which is

          Set drwxrws--- on $job_cache and $dist_cache in all cases. This means user's tasks CAN potentially create unwarranted files/dirs in the $job_cache or $dist-cache or delete their own files themselves.

          So now, the user-directory taskTracker/$user will have "2570 user-owner task-tracker-group" and all the world under it will have "2570/0770 user-owner task-tracker-group".

          Show
          Vinod Kumar Vavilapalli added a comment - Attaching patch which should be applied over the latest patch at MAPREDUCE-856 . It follows a slight modification of the second proposal above, which is Set drwxrws--- on $job_cache and $dist_cache in all cases. This means user's tasks CAN potentially create unwarranted files/dirs in the $job_cache or $dist-cache or delete their own files themselves. So now, the user-directory taskTracker/$user will have "2570 user-owner task-tracker-group" and all the world under it will have "2570/0770 user-owner task-tracker-group".
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Updated patch to be applied over the latest patch at MAPREDUCE-856.

          Show
          Vinod Kumar Vavilapalli added a comment - Updated patch to be applied over the latest patch at MAPREDUCE-856 .
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I think this is a blocker - when running with LinuxTaskController, it prevents job-submission as the user running the mapred daemons.

          Show
          Vinod Kumar Vavilapalli added a comment - I think this is a blocker - when running with LinuxTaskController, it prevents job-submission as the user running the mapred daemons.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          The same (latest) patch works post feature-freeze even with all the patches that went in.

          Show
          Vinod Kumar Vavilapalli added a comment - The same (latest) patch works post feature-freeze even with all the patches that went in.
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12419014/MAPREDUCE-890-20090909.txt
          against trunk revision 834284.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12419014/MAPREDUCE-890-20090909.txt against trunk revision 834284. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/133/console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          The patch above that implements the 2nd proposal needs more changes.

          Set drwxrws--- on $job-dir in all cases. This means user's tasks CAN potentially create unwarranted files/dirs in the $job_dir

          With this, unwarranted files/dirs with improper permissions in job-dir will prevent TT from recursively deleting all the files - a manifestation of MAPREDUCE-896 in the job-dir. This should also be fixed in this patch itself.

          Show
          Vinod Kumar Vavilapalli added a comment - The patch above that implements the 2nd proposal needs more changes. Set drwxrws--- on $job-dir in all cases. This means user's tasks CAN potentially create unwarranted files/dirs in the $job_dir With this, unwarranted files/dirs with improper permissions in job-dir will prevent TT from recursively deleting all the files - a manifestation of MAPREDUCE-896 in the job-dir. This should also be fixed in this patch itself.
          Hide
          Ravi Gummadi added a comment -

          Setting drwxrws--- for jobcache and distcache directories allows users to create files in these directories and as these directories won't be cleaned up for each job(will be cleaned up only when TT is restarted), this can cause space issues.

          So I propose considering tt_user running MR job as a special case and handle that in a special way. i.e. set drwxrwx--- for distcache and jobcache only for jobs of tt_user. This keeps the security of these dirs for all other users as it is there in current trunk.

          Thoughts ?

          Show
          Ravi Gummadi added a comment - Setting drwxrws--- for jobcache and distcache directories allows users to create files in these directories and as these directories won't be cleaned up for each job(will be cleaned up only when TT is restarted), this can cause space issues. So I propose considering tt_user running MR job as a special case and handle that in a special way. i.e. set drwxrwx--- for distcache and jobcache only for jobs of tt_user. This keeps the security of these dirs for all other users as it is there in current trunk. Thoughts ?
          Hide
          Ravi Gummadi added a comment -

          Another solution is to call DefaultTaskController's methods from LinuxTaskContrller's methods if user(i.e. jobOwner) is tt_user. This will keep 700 permissions for userDir, jobcache, distcache dirs and so are secure and are accessible to jobOwner.

          As I don't see any issues with either of these 2 solutions, currently I am planning to implement 1st approach(i.e. LinuxTaskController setting 770 permissions for jobcache and distcache dirs if jobOwner is tt_user).

          Thoughts ?

          Show
          Ravi Gummadi added a comment - Another solution is to call DefaultTaskController's methods from LinuxTaskContrller's methods if user(i.e. jobOwner) is tt_user. This will keep 700 permissions for userDir, jobcache, distcache dirs and so are secure and are accessible to jobOwner. As I don't see any issues with either of these 2 solutions, currently I am planning to implement 1st approach(i.e. LinuxTaskController setting 770 permissions for jobcache and distcache dirs if jobOwner is tt_user). Thoughts ?
          Hide
          Hemanth Yamijala added a comment -

          I would like to give it some more thought. But for now, I think Approach 1 seems to be the right way to go (in spite of the special check). I am assuming the special check is in the task-controller ?

          Show
          Hemanth Yamijala added a comment - I would like to give it some more thought. But for now, I think Approach 1 seems to be the right way to go (in spite of the special check). I am assuming the special check is in the task-controller ?
          Hide
          Ravi Gummadi added a comment -

          Yes Hemanth. Check is in task-controller.

          Show
          Ravi Gummadi added a comment - Yes Hemanth. Check is in task-controller.
          Hide
          Ravi Gummadi added a comment -

          Attaching patch fixing the issue.

          Patch does the following:
          (1) When localizing userDir, in intialize_user(), we do "chmod 2770 -R taskTracker/$user" for tt_user(i.e. if jobOwner is same as tt_user) and "chmod 2570 -R taskTracker/$user" otherwise.

          (2) Similarly,
          (a) "chmod 2770 -R taskTracker/$user/jobcache/$jobid" for tt_user and "chmod 2570 -R taskTracker/$user/jobcache/$jobid" otherwise.
          (b) "chmod 2770 -R taskTracker/$user/distcache/<randomdir>" for tt_user and "chmod 2570 -R taskTracker/$user/distcache/<randomdir>" otherwise.

          (3) compare_ownership() check was done in secure_path() sothat setting of permissions for distributed-cache files(for which permissions are already set earlier) will not go through chown and chmod again and again. This check is not needed after MAPREDUCE-1186(which is already committed to trunk). So deleted this check from secure_path().

          (4) Modified TestLocalizationWithLinuxTaskController sothat validation of permissions of different files/directories will work properly in the cases of (a) jobOwner is tt_user and (b) any other user.

          Please review and provide your comments.

          Show
          Ravi Gummadi added a comment - Attaching patch fixing the issue. Patch does the following: (1) When localizing userDir, in intialize_user(), we do "chmod 2770 -R taskTracker/$user" for tt_user(i.e. if jobOwner is same as tt_user) and "chmod 2570 -R taskTracker/$user" otherwise. (2) Similarly, (a) "chmod 2770 -R taskTracker/$user/jobcache/$jobid" for tt_user and "chmod 2570 -R taskTracker/$user/jobcache/$jobid" otherwise. (b) "chmod 2770 -R taskTracker/$user/distcache/<randomdir>" for tt_user and "chmod 2570 -R taskTracker/$user/distcache/<randomdir>" otherwise. (3) compare_ownership() check was done in secure_path() sothat setting of permissions for distributed-cache files(for which permissions are already set earlier) will not go through chown and chmod again and again. This check is not needed after MAPREDUCE-1186 (which is already committed to trunk). So deleted this check from secure_path(). (4) Modified TestLocalizationWithLinuxTaskController sothat validation of permissions of different files/directories will work properly in the cases of (a) jobOwner is tt_user and (b) any other user. Please review and provide your comments.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Looked at the patch. Got some comments:

          • After patch, at +643, in initialize_user() function, the patch incorrectly passes 0 to secure_path()'s shouldCheckOwnership argument.
          • TestTrackerDistributedCacheManager* tests will need modifications too, I guess.
          • It's already a pain running the task-controller tests. With this patch, the expectation is to run these 5-odd tests with $tt_user and $$non_tt_user, that is two times. I think we should modify the tests themselves so that this double test-run is automatically done.
          Show
          Vinod Kumar Vavilapalli added a comment - Looked at the patch. Got some comments: After patch, at +643, in initialize_user() function, the patch incorrectly passes 0 to secure_path() 's shouldCheckOwnership argument. TestTrackerDistributedCacheManager* tests will need modifications too, I guess. It's already a pain running the task-controller tests. With this patch, the expectation is to run these 5-odd tests with $tt_user and $$non_tt_user, that is two times. I think we should modify the tests themselves so that this double test-run is automatically done.
          Hide
          Ravi Gummadi added a comment -

          As MAPREDUCE-1429 will take care of running the linux task controller based tests with both different user and tt_user(and also changing the source code of all tests is not maintainable — as future linux task controller based testcases should also consciously follow this duplication of testcases for both users), I will drop this(Vinod's 3rd comment) from this patch. Incorporating first 2 comments.

          Show
          Ravi Gummadi added a comment - As MAPREDUCE-1429 will take care of running the linux task controller based tests with both different user and tt_user(and also changing the source code of all tests is not maintainable — as future linux task controller based testcases should also consciously follow this duplication of testcases for both users), I will drop this(Vinod's 3rd comment) from this patch. Incorporating first 2 comments.
          Hide
          Amareshwari Sriramadasu added a comment -

          why don't we run LinuxTaskController tests as the test-launcher(thereby as TT), if nothing is passed in taskcontroller-ugi ?

          Show
          Amareshwari Sriramadasu added a comment - why don't we run LinuxTaskController tests as the test-launcher(thereby as TT), if nothing is passed in taskcontroller-ugi ?
          Hide
          Ravi Gummadi added a comment -

          I don't see much value in changing all LinuxTaskController based tests to take default value for taskcontroller-ugi when taskcontroller-path is specified(anyway user has to specify taskcontroller-path for the tests to be run) and taskcontroller-ugi is not specified.
          Do you really think it is useful ?

          Show
          Ravi Gummadi added a comment - I don't see much value in changing all LinuxTaskController based tests to take default value for taskcontroller-ugi when taskcontroller-path is specified(anyway user has to specify taskcontroller-path for the tests to be run) and taskcontroller-ugi is not specified. Do you really think it is useful ?
          Hide
          Ravi Gummadi added a comment -

          Attaching patch with Vinod's 1st 2 review comments incorporated. I don't see value in making source code changes for 3rd comment and discussed it with Vinod offline and MAPREDUCE-1429 will take care of it, if needed.

          Please review and provide your comments.

          Show
          Ravi Gummadi added a comment - Attaching patch with Vinod's 1st 2 review comments incorporated. I don't see value in making source code changes for 3rd comment and discussed it with Vinod offline and MAPREDUCE-1429 will take care of it, if needed. Please review and provide your comments.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          The patch is not applying over the latest patch at MAPREDUCE-1421 which is a blocker for this JIRA issue. Can you please upload an updated patch?

          Show
          Vinod Kumar Vavilapalli added a comment - The patch is not applying over the latest patch at MAPREDUCE-1421 which is a blocker for this JIRA issue. Can you please upload an updated patch?
          Hide
          Ravi Gummadi added a comment -

          Syncing the patch with the current trunk.

          Show
          Ravi Gummadi added a comment - Syncing the patch with the current trunk.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Patch looks good to me. +1, pending MAPREDUCE-1421.

          Show
          Vinod Kumar Vavilapalli added a comment - Patch looks good to me. +1, pending MAPREDUCE-1421 .
          Hide
          Vinod Kumar Vavilapalli added a comment -

          MAPREDUCE-1421 is committed. Running this patch through Hudson.

          Show
          Vinod Kumar Vavilapalli added a comment - MAPREDUCE-1421 is committed. Running this patch through Hudson.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Ravi, can you run tests and test-patch? Hudson is not picking patches up, so..

          Show
          Vinod Kumar Vavilapalli added a comment - Ravi, can you run tests and test-patch? Hudson is not picking patches up, so..
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12437589/MR890.v1.1.patch
          against trunk revision 918864.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 24 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437589/MR890.v1.1.patch against trunk revision 918864. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/342/console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          The tests TestSymLink.testSymLink, TestUlimit, TestIO failed because of the test run by Hudson could not find some of the core class. Wierd. Anyone knows why?

          Filed MAPREDUCE-1562 for the failure of TestBadRecords.testBadMapRed.

          Show
          Vinod Kumar Vavilapalli added a comment - The tests TestSymLink.testSymLink, TestUlimit, TestIO failed because of the test run by Hudson could not find some of the core class. Wierd. Anyone knows why? Filed MAPREDUCE-1562 for the failure of TestBadRecords.testBadMapRed.
          Hide
          Ravi Gummadi added a comment -

          Syncing with the latest trunk(as MAPREDUCE-1435 went in).

          Show
          Ravi Gummadi added a comment - Syncing with the latest trunk(as MAPREDUCE-1435 went in).
          Hide
          Ravi Gummadi added a comment -

          Allowing it to go through Hudson

          Show
          Ravi Gummadi added a comment - Allowing it to go through Hudson
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12437984/MR890.v1.2.patch
          against trunk revision 919335.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 24 new or modified tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12437984/MR890.v1.2.patch against trunk revision 919335. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/23/console This message is automatically generated.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          For some reason, Hudson's couldn't report which tests failed on its web portal, even though the console output is still there. I've downloaded the console output and saw that all the tests passed except TestTaskTrackerMemoryManager. And this failure is because of the same wierd java.lang.NoClassDefFoundError on some core classes that's been plaguing other test runs, this time on org.apache.hadoop.http.HttpServer$QuotingInputFilter.

          I could run TestTaskTrackerMemoryManager successfully. I am going to check this in.

          Show
          Vinod Kumar Vavilapalli added a comment - For some reason, Hudson's couldn't report which tests failed on its web portal, even though the console output is still there. I've downloaded the console output and saw that all the tests passed except TestTaskTrackerMemoryManager. And this failure is because of the same wierd java.lang.NoClassDefFoundError on some core classes that's been plaguing other test runs, this time on org.apache.hadoop.http.HttpServer$QuotingInputFilter. I could run TestTaskTrackerMemoryManager successfully. I am going to check this in.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I've just committed this to trunk. Thanks Ravi!

          The trunk patch isn't applying to 0.21 branch. I think we'll need a new one, can you work on it?

          Show
          Vinod Kumar Vavilapalli added a comment - I've just committed this to trunk. Thanks Ravi! The trunk patch isn't applying to 0.21 branch. I think we'll need a new one, can you work on it?
          Hide
          Ravi Gummadi added a comment -

          Attaching patch for earlier version of hadoop(includes the fix of MAPREDUCE-1573 also). Not for commit here.

          Show
          Ravi Gummadi added a comment - Attaching patch for earlier version of hadoop(includes the fix of MAPREDUCE-1573 also). Not for commit here.
          Hide
          Chris Douglas added a comment -

          Since this was already committed to trunk, I'm going to mark it as resolved purely for accounting purposes. I've also moved the entry from 0.21 to 0.22 in CHANGES.txt. Once the patch for 0.21 is committed, the entry can be moved back to 0.21.

          Show
          Chris Douglas added a comment - Since this was already committed to trunk, I'm going to mark it as resolved purely for accounting purposes. I've also moved the entry from 0.21 to 0.22 in CHANGES.txt. Once the patch for 0.21 is committed, the entry can be moved back to 0.21.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #285 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/285/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #285 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/285/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk #264 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/264/)
          Move from 0.21 to 0.22 in changelog to match commit

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #264 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Mapreduce-trunk/264/ ) Move from 0.21 to 0.22 in changelog to match commit

            People

            • Assignee:
              Ravi Gummadi
              Reporter:
              Karam Singh
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development