Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.21.0
    • Component/s: documentation, security
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Job acls introduced in MAPREDUCE-1307 should be documented in forrest.

      1. patch-1604.txt
        5 kB
        Amareshwari Sriramadasu
      2. patch-1604-1.txt
        5 kB
        Amareshwari Sriramadasu
      3. patch-1604-ydist.txt
        5 kB
        Amareshwari Sriramadasu

        Activity

        Hide
        Vinod Kumar Vavilapalli added a comment -

        No, I don't object. If you feel that's the right call that will simplify life in future, please go ahead.

        Thanks Hemanth, I opened MAPREDUCE-1747 for removing the documentation of job-acls for the sake of release 0.21 and MAPREDUCE-1748 for putting it back into trunk/0.22.

        Show
        Vinod Kumar Vavilapalli added a comment - No, I don't object. If you feel that's the right call that will simplify life in future, please go ahead. Thanks Hemanth, I opened MAPREDUCE-1747 for removing the documentation of job-acls for the sake of release 0.21 and MAPREDUCE-1748 for putting it back into trunk/0.22.
        Hide
        Hemanth Yamijala added a comment -

        No, I don't object. If you feel that's the right call that will simplify life in future, please go ahead.

        Show
        Hemanth Yamijala added a comment - No, I don't object. If you feel that's the right call that will simplify life in future, please go ahead.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        I think we are assuming a lot of people use queues and queue ACLs. How true is that ? The default scheduler and the fair scheduler don't support queues. Isn't that a lot of users ? Wouldn't it mean they'd simply ignore queue ACLs and then job ACLs actually work fine, as documented ?

        Missed this one, and the freeze date is right here.

        Irrespective of the number of users using job-acls, post MAPREDUCE-1664, job-authorization.enabled flag will be either removed or deprecated in favour of mapred-queues.enabled flag. Irrespecitve of whether it is removed or deprecated, it will no longer be backwards compatible because semantics of the whole ACLs across queues and jobs are going to change. Given this, I am still inclined to removing the documentation for job-acls from 0.21. Unless you veto this, of course Hemanth, can you quickly respond?

        Show
        Vinod Kumar Vavilapalli added a comment - I think we are assuming a lot of people use queues and queue ACLs. How true is that ? The default scheduler and the fair scheduler don't support queues. Isn't that a lot of users ? Wouldn't it mean they'd simply ignore queue ACLs and then job ACLs actually work fine, as documented ? Missed this one, and the freeze date is right here. Irrespective of the number of users using job-acls, post MAPREDUCE-1664 , job-authorization.enabled flag will be either removed or deprecated in favour of mapred-queues.enabled flag. Irrespecitve of whether it is removed or deprecated, it will no longer be backwards compatible because semantics of the whole ACLs across queues and jobs are going to change. Given this, I am still inclined to removing the documentation for job-acls from 0.21. Unless you veto this, of course Hemanth, can you quickly respond?
        Hide
        Hemanth Yamijala added a comment -

        I think we are assuming a lot of people use queues and queue ACLs. How true is that ? The default scheduler and the fair scheduler don't support queues. Isn't that a lot of users ? Wouldn't it mean they'd simply ignore queue ACLs and then job ACLs actually work fine, as documented ?

        Show
        Hemanth Yamijala added a comment - I think we are assuming a lot of people use queues and queue ACLs. How true is that ? The default scheduler and the fair scheduler don't support queues. Isn't that a lot of users ? Wouldn't it mean they'd simply ignore queue ACLs and then job ACLs actually work fine, as documented ?
        Hide
        Vinod Kumar Vavilapalli added a comment -

        I agree with your comments about documentation being part of the whole.

        My only problem here is Job-acls ARE an unstable feature. The moment one configures job-acls to be used, issues like MAPREDUCE-1664 crop up. Also if MAPREDUCE-1664 doesn't go in 0.21, we will need to support job-authorization.enabled flag in 0.22 as it is going away in favour of mapred-acls-enabled flag.

        I am just trying to see if we should remove the documentation of this feature for release 0.21 and get the complete feature in for 0.22. I am not sure of any process via which we can tell users about known issues. Even if we have one, I'd rather prefer turning off this feature completely in 0.21 as without MAPREDUCE-1664 and related issues, this issue is practically useless. What do you think about it?

        Show
        Vinod Kumar Vavilapalli added a comment - I agree with your comments about documentation being part of the whole. My only problem here is Job-acls ARE an unstable feature. The moment one configures job-acls to be used, issues like MAPREDUCE-1664 crop up. Also if MAPREDUCE-1664 doesn't go in 0.21, we will need to support job-authorization.enabled flag in 0.22 as it is going away in favour of mapred-acls-enabled flag. I am just trying to see if we should remove the documentation of this feature for release 0.21 and get the complete feature in for 0.22. I am not sure of any process via which we can tell users about known issues. Even if we have one, I'd rather prefer turning off this feature completely in 0.21 as without MAPREDUCE-1664 and related issues, this issue is practically useless. What do you think about it?
        Hide
        Hemanth Yamijala added a comment -

        Vinod, I was aware there are issues being worked on. In general, a feature is documented with a patch. In that sense, this JIRA simply made sure that the feature in code is documented in Forrest. Is there anything that was documented that was incorrect ? If yes, that's a bug in documentation and should be fixed. If everything written was technically correct, but the feature has gotchas itself, I suppose one option would be to call out the gotchas in the documentation and thus make it more user-friendly and warn users about potential issues. AFAIK, the adjustments being made are not going to drastically change what has been documented. So most of the documentation should still hold. No ?

        Show
        Hemanth Yamijala added a comment - Vinod, I was aware there are issues being worked on. In general, a feature is documented with a patch. In that sense, this JIRA simply made sure that the feature in code is documented in Forrest. Is there anything that was documented that was incorrect ? If yes, that's a bug in documentation and should be fixed. If everything written was technically correct, but the feature has gotchas itself, I suppose one option would be to call out the gotchas in the documentation and thus make it more user-friendly and warn users about potential issues. AFAIK, the adjustments being made are not going to drastically change what has been documented. So most of the documentation should still hold. No ?
        Hide
        Vinod Kumar Vavilapalli added a comment -

        Argh.. I was trying to see if we should hold of this commit. I just woke up to realize that trunk is being rebased as release 0.21. Given this, and the fact that the job-acls feature a.k.a MAPREDUCE-1307 is still only 'half-complete' because of MAPREDUCE-1664 and other related minor issues. I checked with Ravi who started working on MAPREDUCE-1664 and came to know that it will not be possible to get it in by freeze date(30th?) because it involves some consensus building as well as some time for implementation.

        Two things can be done. Complete MAPREDUCE-1664 by marking it as a blocker. For this we will need to know the release time lines for 0.21. Or consider job-acls as an unstable feature and undocument it from mapred-default.xml and forrest. The later is a plausibility not just because of MAPREDUCE-1664, but also due to other subsequent issues that arose out of this feature that we found out in internal discussions/testing. Thoughts?

        BTW, there must be a lot of 'half-features' like these which are originally being worked upon for 0.22 but will fall back into 0.21 because of the recent decision. For e.g., HADOOP-6568 which has become 'unstable' due to uncertainty of MAPREDUCE-1542 and HDFS-1008.

        What is the community thinking about these issues? Will need to shoot an email on general mailing list, I guess.

        Show
        Vinod Kumar Vavilapalli added a comment - Argh.. I was trying to see if we should hold of this commit. I just woke up to realize that trunk is being rebased as release 0.21. Given this, and the fact that the job-acls feature a.k.a MAPREDUCE-1307 is still only 'half-complete' because of MAPREDUCE-1664 and other related minor issues. I checked with Ravi who started working on MAPREDUCE-1664 and came to know that it will not be possible to get it in by freeze date(30th?) because it involves some consensus building as well as some time for implementation. Two things can be done. Complete MAPREDUCE-1664 by marking it as a blocker. For this we will need to know the release time lines for 0.21. Or consider job-acls as an unstable feature and undocument it from mapred-default.xml and forrest. The later is a plausibility not just because of MAPREDUCE-1664 , but also due to other subsequent issues that arose out of this feature that we found out in internal discussions/testing. Thoughts? BTW, there must be a lot of 'half-features' like these which are originally being worked upon for 0.22 but will fall back into 0.21 because of the recent decision. For e.g., HADOOP-6568 which has become 'unstable' due to uncertainty of MAPREDUCE-1542 and HDFS-1008 . What is the community thinking about these issues? Will need to shoot an email on general mailing list, I guess.
        Hide
        Hemanth Yamijala added a comment -

        I just committed this. Thanks, Amareshwari !

        Show
        Hemanth Yamijala added a comment - I just committed this. Thanks, Amareshwari !
        Hide
        Hemanth Yamijala added a comment -

        The core test failures are unrelated and are being tracked in MAPREDUCE-1727. The +0 on tests is because this is a documentation patch. This is ready to go.

        Show
        Hemanth Yamijala added a comment - The core test failures are unrelated and are being tracked in MAPREDUCE-1727 . The +0 on tests is because this is a documentation patch. This is ready to go.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12442831/patch-1604-1.txt
        against trunk revision 937924.

        +1 @author. The patch does not contain any @author tags.

        +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12442831/patch-1604-1.txt against trunk revision 937924. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/console This message is automatically generated.
        Hide
        Hemanth Yamijala added a comment -

        The latest patch looks good to me. I will commit after Hudson returns.

        Show
        Hemanth Yamijala added a comment - The latest patch looks good to me. I will commit after Hudson returns.
        Hide
        Amareshwari Sriramadasu added a comment -

        ant docs ran successfully with the patch.

        Show
        Amareshwari Sriramadasu added a comment - ant docs ran successfully with the patch.
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch updated to trunk.

        Show
        Amareshwari Sriramadasu added a comment - Patch updated to trunk.
        Hide
        Vinod Kumar Vavilapalli added a comment -

        It is not applying anymore, can you please update the patch?

        Show
        Vinod Kumar Vavilapalli added a comment - It is not applying anymore, can you please update the patch?
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12439115/patch-1604.txt
        against trunk revision 923907.

        +1 @author. The patch does not contain any @author tags.

        +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/testReport/
        Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/artifact/trunk/build/test/checkstyle-errors.html
        Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12439115/patch-1604.txt against trunk revision 923907. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/console This message is automatically generated.
        Hide
        Amareshwari Sriramadasu added a comment -

        Attaching the patch for trunk.

        ant docs passed successfully on my machine.

        Show
        Amareshwari Sriramadasu added a comment - Attaching the patch for trunk. ant docs passed successfully on my machine.
        Hide
        Amareshwari Sriramadasu added a comment -

        Patch for Yahoo! distribution

        Show
        Amareshwari Sriramadasu added a comment - Patch for Yahoo! distribution

          People

          • Assignee:
            Amareshwari Sriramadasu
            Reporter:
            Amareshwari Sriramadasu
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development