Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.22.0
    • 0.21.0
    • documentation, security
    • None
    • Reviewed

    Description

      Job acls introduced in MAPREDUCE-1307 should be documented in forrest.

      Attachments

        1. patch-1604.txt
          5 kB
          Amareshwari Sriramadasu
        2. patch-1604-1.txt
          5 kB
          Amareshwari Sriramadasu
        3. patch-1604-ydist.txt
          5 kB
          Amareshwari Sriramadasu

        Activity

          Patch for Yahoo! distribution

          amareshwari Amareshwari Sriramadasu added a comment - Patch for Yahoo! distribution

          Attaching the patch for trunk.

          ant docs passed successfully on my machine.

          amareshwari Amareshwari Sriramadasu added a comment - Attaching the patch for trunk. ant docs passed successfully on my machine.
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12439115/patch-1604.txt
          against trunk revision 923907.

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12439115/patch-1604.txt against trunk revision 923907. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/529/console This message is automatically generated.

          It is not applying anymore, can you please update the patch?

          vinodkv Vinod Kumar Vavilapalli added a comment - It is not applying anymore, can you please update the patch?

          Patch updated to trunk.

          amareshwari Amareshwari Sriramadasu added a comment - Patch updated to trunk.

          ant docs ran successfully with the patch.

          amareshwari Amareshwari Sriramadasu added a comment - ant docs ran successfully with the patch.

          The latest patch looks good to me. I will commit after Hudson returns.

          yhemanth Hemanth Yamijala added a comment - The latest patch looks good to me. I will commit after Hudson returns.
          hadoopqa Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12442831/patch-1604-1.txt
          against trunk revision 937924.

          +1 @author. The patch does not contain any @author tags.

          +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/console

          This message is automatically generated.

          hadoopqa Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12442831/patch-1604-1.txt against trunk revision 937924. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h3.grid.sp2.yahoo.net/361/console This message is automatically generated.

          The core test failures are unrelated and are being tracked in MAPREDUCE-1727. The +0 on tests is because this is a documentation patch. This is ready to go.

          yhemanth Hemanth Yamijala added a comment - The core test failures are unrelated and are being tracked in MAPREDUCE-1727 . The +0 on tests is because this is a documentation patch. This is ready to go.

          I just committed this. Thanks, Amareshwari !

          yhemanth Hemanth Yamijala added a comment - I just committed this. Thanks, Amareshwari !

          Argh.. I was trying to see if we should hold of this commit. I just woke up to realize that trunk is being rebased as release 0.21. Given this, and the fact that the job-acls feature a.k.a MAPREDUCE-1307 is still only 'half-complete' because of MAPREDUCE-1664 and other related minor issues. I checked with Ravi who started working on MAPREDUCE-1664 and came to know that it will not be possible to get it in by freeze date(30th?) because it involves some consensus building as well as some time for implementation.

          Two things can be done. Complete MAPREDUCE-1664 by marking it as a blocker. For this we will need to know the release time lines for 0.21. Or consider job-acls as an unstable feature and undocument it from mapred-default.xml and forrest. The later is a plausibility not just because of MAPREDUCE-1664, but also due to other subsequent issues that arose out of this feature that we found out in internal discussions/testing. Thoughts?

          BTW, there must be a lot of 'half-features' like these which are originally being worked upon for 0.22 but will fall back into 0.21 because of the recent decision. For e.g., HADOOP-6568 which has become 'unstable' due to uncertainty of MAPREDUCE-1542 and HDFS-1008.

          What is the community thinking about these issues? Will need to shoot an email on general mailing list, I guess.

          vinodkv Vinod Kumar Vavilapalli added a comment - Argh.. I was trying to see if we should hold of this commit. I just woke up to realize that trunk is being rebased as release 0.21. Given this, and the fact that the job-acls feature a.k.a MAPREDUCE-1307 is still only 'half-complete' because of MAPREDUCE-1664 and other related minor issues. I checked with Ravi who started working on MAPREDUCE-1664 and came to know that it will not be possible to get it in by freeze date(30th?) because it involves some consensus building as well as some time for implementation. Two things can be done. Complete MAPREDUCE-1664 by marking it as a blocker. For this we will need to know the release time lines for 0.21. Or consider job-acls as an unstable feature and undocument it from mapred-default.xml and forrest. The later is a plausibility not just because of MAPREDUCE-1664 , but also due to other subsequent issues that arose out of this feature that we found out in internal discussions/testing. Thoughts? BTW, there must be a lot of 'half-features' like these which are originally being worked upon for 0.22 but will fall back into 0.21 because of the recent decision. For e.g., HADOOP-6568 which has become 'unstable' due to uncertainty of MAPREDUCE-1542 and HDFS-1008 . What is the community thinking about these issues? Will need to shoot an email on general mailing list, I guess.

          Vinod, I was aware there are issues being worked on. In general, a feature is documented with a patch. In that sense, this JIRA simply made sure that the feature in code is documented in Forrest. Is there anything that was documented that was incorrect ? If yes, that's a bug in documentation and should be fixed. If everything written was technically correct, but the feature has gotchas itself, I suppose one option would be to call out the gotchas in the documentation and thus make it more user-friendly and warn users about potential issues. AFAIK, the adjustments being made are not going to drastically change what has been documented. So most of the documentation should still hold. No ?

          yhemanth Hemanth Yamijala added a comment - Vinod, I was aware there are issues being worked on. In general, a feature is documented with a patch. In that sense, this JIRA simply made sure that the feature in code is documented in Forrest. Is there anything that was documented that was incorrect ? If yes, that's a bug in documentation and should be fixed. If everything written was technically correct, but the feature has gotchas itself, I suppose one option would be to call out the gotchas in the documentation and thus make it more user-friendly and warn users about potential issues. AFAIK, the adjustments being made are not going to drastically change what has been documented. So most of the documentation should still hold. No ?

          I agree with your comments about documentation being part of the whole.

          My only problem here is Job-acls ARE an unstable feature. The moment one configures job-acls to be used, issues like MAPREDUCE-1664 crop up. Also if MAPREDUCE-1664 doesn't go in 0.21, we will need to support job-authorization.enabled flag in 0.22 as it is going away in favour of mapred-acls-enabled flag.

          I am just trying to see if we should remove the documentation of this feature for release 0.21 and get the complete feature in for 0.22. I am not sure of any process via which we can tell users about known issues. Even if we have one, I'd rather prefer turning off this feature completely in 0.21 as without MAPREDUCE-1664 and related issues, this issue is practically useless. What do you think about it?

          vinodkv Vinod Kumar Vavilapalli added a comment - I agree with your comments about documentation being part of the whole. My only problem here is Job-acls ARE an unstable feature. The moment one configures job-acls to be used, issues like MAPREDUCE-1664 crop up. Also if MAPREDUCE-1664 doesn't go in 0.21, we will need to support job-authorization.enabled flag in 0.22 as it is going away in favour of mapred-acls-enabled flag. I am just trying to see if we should remove the documentation of this feature for release 0.21 and get the complete feature in for 0.22. I am not sure of any process via which we can tell users about known issues. Even if we have one, I'd rather prefer turning off this feature completely in 0.21 as without MAPREDUCE-1664 and related issues, this issue is practically useless. What do you think about it?

          I think we are assuming a lot of people use queues and queue ACLs. How true is that ? The default scheduler and the fair scheduler don't support queues. Isn't that a lot of users ? Wouldn't it mean they'd simply ignore queue ACLs and then job ACLs actually work fine, as documented ?

          yhemanth Hemanth Yamijala added a comment - I think we are assuming a lot of people use queues and queue ACLs. How true is that ? The default scheduler and the fair scheduler don't support queues. Isn't that a lot of users ? Wouldn't it mean they'd simply ignore queue ACLs and then job ACLs actually work fine, as documented ?

          I think we are assuming a lot of people use queues and queue ACLs. How true is that ? The default scheduler and the fair scheduler don't support queues. Isn't that a lot of users ? Wouldn't it mean they'd simply ignore queue ACLs and then job ACLs actually work fine, as documented ?

          Missed this one, and the freeze date is right here.

          Irrespective of the number of users using job-acls, post MAPREDUCE-1664, job-authorization.enabled flag will be either removed or deprecated in favour of mapred-queues.enabled flag. Irrespecitve of whether it is removed or deprecated, it will no longer be backwards compatible because semantics of the whole ACLs across queues and jobs are going to change. Given this, I am still inclined to removing the documentation for job-acls from 0.21. Unless you veto this, of course Hemanth, can you quickly respond?

          vinodkv Vinod Kumar Vavilapalli added a comment - I think we are assuming a lot of people use queues and queue ACLs. How true is that ? The default scheduler and the fair scheduler don't support queues. Isn't that a lot of users ? Wouldn't it mean they'd simply ignore queue ACLs and then job ACLs actually work fine, as documented ? Missed this one, and the freeze date is right here. Irrespective of the number of users using job-acls, post MAPREDUCE-1664 , job-authorization.enabled flag will be either removed or deprecated in favour of mapred-queues.enabled flag. Irrespecitve of whether it is removed or deprecated, it will no longer be backwards compatible because semantics of the whole ACLs across queues and jobs are going to change. Given this, I am still inclined to removing the documentation for job-acls from 0.21. Unless you veto this, of course Hemanth, can you quickly respond?

          No, I don't object. If you feel that's the right call that will simplify life in future, please go ahead.

          yhemanth Hemanth Yamijala added a comment - No, I don't object. If you feel that's the right call that will simplify life in future, please go ahead.

          No, I don't object. If you feel that's the right call that will simplify life in future, please go ahead.

          Thanks Hemanth, I opened MAPREDUCE-1747 for removing the documentation of job-acls for the sake of release 0.21 and MAPREDUCE-1748 for putting it back into trunk/0.22.

          vinodkv Vinod Kumar Vavilapalli added a comment - No, I don't object. If you feel that's the right call that will simplify life in future, please go ahead. Thanks Hemanth, I opened MAPREDUCE-1747 for removing the documentation of job-acls for the sake of release 0.21 and MAPREDUCE-1748 for putting it back into trunk/0.22.

          People

            amareshwari Amareshwari Sriramadasu
            amareshwari Amareshwari Sriramadasu
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: