Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: documentation, security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Job acls introduced in MAPREDUCE-1307 should be documented in forrest.

      1. patch-1604.txt
        5 kB
        Amareshwari Sriramadasu
      2. patch-1604-1.txt
        5 kB
        Amareshwari Sriramadasu
      3. patch-1604-ydist.txt
        5 kB
        Amareshwari Sriramadasu

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Patch Available Patch Available Open Open
        39d 2h 13m 1 Vinod Kumar Vavilapalli 26/Apr/10 08:38
        Open Open Patch Available Patch Available
        23h 59m 2 Amareshwari Sriramadasu 26/Apr/10 09:16
        Patch Available Patch Available Resolved Resolved
        4h 26m 1 Hemanth Yamijala 26/Apr/10 13:42
        Resolved Resolved Closed Closed
        120d 8h 38m 1 Tom White 24/Aug/10 22:21
        Tom White made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Tom White made changes -
        Fix Version/s 0.21.0 [ 12314045 ]
        Fix Version/s 0.22.0 [ 12314184 ]
        Hide
        Vinod Kumar Vavilapalli added a comment -

        No, I don't object. If you feel that's the right call that will simplify life in future, please go ahead.

        Thanks Hemanth, I opened MAPREDUCE-1747 for removing the documentation of job-acls for the sake of release 0.21 and MAPREDUCE-1748 for putting it back into trunk/0.22.

        Show
        Vinod Kumar Vavilapalli added a comment - No, I don't object. If you feel that's the right call that will simplify life in future, please go ahead. Thanks Hemanth, I opened MAPREDUCE-1747 for removing the documentation of job-acls for the sake of release 0.21 and MAPREDUCE-1748 for putting it back into trunk/0.22.
        Hide
        Hemanth Yamijala added a comment -

        No, I don't object. If you feel that's the right call that will simplify life in future, please go ahead.

        Show
        Hemanth Yamijala added a comment - No, I don't object. If you feel that's the right call that will simplify life in future, please go ahead.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        I think we are assuming a lot of people use queues and queue ACLs. How true is that ? The default scheduler and the fair scheduler don't support queues. Isn't that a lot of users ? Wouldn't it mean they'd simply ignore queue ACLs and then job ACLs actually work fine, as documented ?

        Missed this one, and the freeze date is right here.

        Irrespective of the number of users using job-acls, post MAPREDUCE-1664, job-authorization.enabled flag will be either removed or deprecated in favour of mapred-queues.enabled flag. Irrespecitve of whether it is removed or deprecated, it will no longer be backwards compatible because semantics of the whole ACLs across queues and jobs are going to change. Given this, I am still inclined to removing the documentation for job-acls from 0.21. Unless you veto this, of course Hemanth, can you quickly respond?

        Show
        Vinod Kumar Vavilapalli added a comment - I think we are assuming a lot of people use queues and queue ACLs. How true is that ? The default scheduler and the fair scheduler don't support queues. Isn't that a lot of users ? Wouldn't it mean they'd simply ignore queue ACLs and then job ACLs actually work fine, as documented ? Missed this one, and the freeze date is right here. Irrespective of the number of users using job-acls, post MAPREDUCE-1664 , job-authorization.enabled flag will be either removed or deprecated in favour of mapred-queues.enabled flag. Irrespecitve of whether it is removed or deprecated, it will no longer be backwards compatible because semantics of the whole ACLs across queues and jobs are going to change. Given this, I am still inclined to removing the documentation for job-acls from 0.21. Unless you veto this, of course Hemanth, can you quickly respond?
        Hide
        Hemanth Yamijala added a comment -

        I think we are assuming a lot of people use queues and queue ACLs. How true is that ? The default scheduler and the fair scheduler don't support queues. Isn't that a lot of users ? Wouldn't it mean they'd simply ignore queue ACLs and then job ACLs actually work fine, as documented ?

        Show
        Hemanth Yamijala added a comment - I think we are assuming a lot of people use queues and queue ACLs. How true is that ? The default scheduler and the fair scheduler don't support queues. Isn't that a lot of users ? Wouldn't it mean they'd simply ignore queue ACLs and then job ACLs actually work fine, as documented ?
        Hide
        Vinod Kumar Vavilapalli added a comment -

        I agree with your comments about documentation being part of the whole.

        My only problem here is Job-acls ARE an unstable feature. The moment one configures job-acls to be used, issues like MAPREDUCE-1664 crop up. Also if MAPREDUCE-1664 doesn't go in 0.21, we will need to support job-authorization.enabled flag in 0.22 as it is going away in favour of mapred-acls-enabled flag.

        I am just trying to see if we should remove the documentation of this feature for release 0.21 and get the complete feature in for 0.22. I am not sure of any process via which we can tell users about known issues. Even if we have one, I'd rather prefer turning off this feature completely in 0.21 as without MAPREDUCE-1664 and related issues, this issue is practically useless. What do you think about it?

        Show
        Vinod Kumar Vavilapalli added a comment - I agree with your comments about documentation being part of the whole. My only problem here is Job-acls ARE an unstable feature. The moment one configures job-acls to be used, issues like MAPREDUCE-1664 crop up. Also if MAPREDUCE-1664 doesn't go in 0.21, we will need to support job-authorization.enabled flag in 0.22 as it is going away in favour of mapred-acls-enabled flag. I am just trying to see if we should remove the documentation of this feature for release 0.21 and get the complete feature in for 0.22. I am not sure of any process via which we can tell users about known issues. Even if we have one, I'd rather prefer turning off this feature completely in 0.21 as without MAPREDUCE-1664 and related issues, this issue is practically useless. What do you think about it?
        Hide
        Hemanth Yamijala added a comment -

        Vinod, I was aware there are issues being worked on. In general, a feature is documented with a patch. In that sense, this JIRA simply made sure that the feature in code is documented in Forrest. Is there anything that was documented that was incorrect ? If yes, that's a bug in documentation and should be fixed. If everything written was technically correct, but the feature has gotchas itself, I suppose one option would be to call out the gotchas in the documentation and thus make it more user-friendly and warn users about potential issues. AFAIK, the adjustments being made are not going to drastically change what has been documented. So most of the documentation should still hold. No ?

        Show
        Hemanth Yamijala added a comment - Vinod, I was aware there are issues being worked on. In general, a feature is documented with a patch. In that sense, this JIRA simply made sure that the feature in code is documented in Forrest. Is there anything that was documented that was incorrect ? If yes, that's a bug in documentation and should be fixed. If everything written was technically correct, but the feature has gotchas itself, I suppose one option would be to call out the gotchas in the documentation and thus make it more user-friendly and warn users about potential issues. AFAIK, the adjustments being made are not going to drastically change what has been documented. So most of the documentation should still hold. No ?
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Argh.. I was trying to see if we should hold of this commit. I just woke up to realize that trunk is being rebased as release 0.21. Given this, and the fact that the job-acls feature a.k.a MAPREDUCE-1307 is still only 'half-complete' because of MAPREDUCE-1664 and other related minor issues. I checked with Ravi who started working on MAPREDUCE-1664 and came to know that it will not be possible to get it in by freeze date(30th?) because it involves some consensus building as well as some time for implementation.

        Two things can be done. Complete MAPREDUCE-1664 by marking it as a blocker. For this we will need to know the release time lines for 0.21. Or consider job-acls as an unstable feature and undocument it from mapred-default.xml and forrest. The later is a plausibility not just because of MAPREDUCE-1664, but also due to other subsequent issues that arose out of this feature that we found out in internal discussions/testing. Thoughts?

        BTW, there must be a lot of 'half-features' like these which are originally being worked upon for 0.22 but will fall back into 0.21 because of the recent decision. For e.g., HADOOP-6568 which has become 'unstable' due to uncertainty of MAPREDUCE-1542 and HDFS-1008.

        What is the community thinking about these issues? Will need to shoot an email on general mailing list, I guess.

        Show
        Vinod Kumar Vavilapalli added a comment - Argh.. I was trying to see if we should hold of this commit. I just woke up to realize that trunk is being rebased as release 0.21. Given this, and the fact that the job-acls feature a.k.a MAPREDUCE-1307 is still only 'half-complete' because of MAPREDUCE-1664 and other related minor issues. I checked with Ravi who started working on MAPREDUCE-1664 and came to know that it will not be possible to get it in by freeze date(30th?) because it involves some consensus building as well as some time for implementation. Two things can be done. Complete MAPREDUCE-1664 by marking it as a blocker. For this we will need to know the release time lines for 0.21. Or consider job-acls as an unstable feature and undocument it from mapred-default.xml and forrest. The later is a plausibility not just because of MAPREDUCE-1664 , but also due to other subsequent issues that arose out of this feature that we found out in internal discussions/testing. Thoughts? BTW, there must be a lot of 'half-features' like these which are originally being worked upon for 0.22 but will fall back into 0.21 because of the recent decision. For e.g., HADOOP-6568 which has become 'unstable' due to uncertainty of MAPREDUCE-1542 and HDFS-1008 . What is the community thinking about these issues? Will need to shoot an email on general mailing list, I guess.
        Hemanth Yamijala made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Resolution Fixed [ 1 ]
        Hide
        Hemanth Yamijala added a comment -

        I just committed this. Thanks, Amareshwari !

        Show
        Hemanth Yamijala added a comment - I just committed this. Thanks, Amareshwari !
        Hide
        Hemanth Yamijala added a comment -

        The core test failures are unrelated and are being tracked in MAPREDUCE-1727. The +0 on tests is because this is a documentation patch. This is ready to go.

        Show
        Hemanth Yamijala added a comment - The core test failures are unrelated and are being tracked in MAPREDUCE-1727 . The +0 on tests is because this is a documentation patch. This is ready to go.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12442831/patch-1604-1.txt
        against trunk revision 937924.

        +1 @author. The patch does not contain any @author tags.

        +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12442831/patch-1604-1.txt against trunk revision 937924. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/console This message is automatically generated.
        Hide
        Hemanth Yamijala added a comment -

        The latest patch looks good to me. I will commit after Hudson returns.

        Show
        Hemanth Yamijala added a comment - The latest patch looks good to me. I will commit after Hudson returns.
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Amareshwari Sriramadasu added a comment -

        ant docs ran successfully with the patch.

        Show
        Amareshwari Sriramadasu added a comment - ant docs ran successfully with the patch.
        Amareshwari Sriramadasu made changes -
        Attachment patch-1604-1.txt [ 12442831 ]
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch updated to trunk.

        Show
        Amareshwari Sriramadasu added a comment - Patch updated to trunk.
        Vinod Kumar Vavilapalli made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Hide
        Vinod Kumar Vavilapalli added a comment -

        It is not applying anymore, can you please update the patch?

        Show
        Vinod Kumar Vavilapalli added a comment - It is not applying anymore, can you please update the patch?
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12439115/patch-1604.txt
        against trunk revision 923907.

        +1 @author. The patch does not contain any @author tags.

        +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12439115/patch-1604.txt against trunk revision 923907. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/console This message is automatically generated.
        Amareshwari Sriramadasu made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Assignee Amareshwari Sriramadasu [ amareshwari ]
        Amareshwari Sriramadasu made changes -
        Attachment patch-1604.txt [ 12439115 ]
        Hide
        Amareshwari Sriramadasu added a comment -

        Attaching the patch for trunk.

        ant docs passed successfully on my machine.

        Show
        Amareshwari Sriramadasu added a comment - Attaching the patch for trunk. ant docs passed successfully on my machine.
        Amareshwari Sriramadasu made changes -
        Field Original Value New Value
        Attachment patch-1604-ydist.txt [ 12439114 ]
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch for Yahoo! distribution

        Show
        Amareshwari Sriramadasu added a comment - Patch for Yahoo! distribution
        Amareshwari Sriramadasu created issue -

          People

          • Assignee:
            Amareshwari Sriramadasu
            Reporter:
            Amareshwari Sriramadasu
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development