Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2505

Explain how to use ACLs in the fair scheduler

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.2, 0.20.203.0, 0.21.0, 0.22.0
    • Fix Version/s: 0.22.0
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The fair scheduler already works with the ACL system introduced through the mapred.queue.* parameters, but the documentation doesn't explain how to use this. We should add a paragraph or two about it.

      1. MAPREDUCE-2505-branch-0.20-security.patch
        3 kB
        Matei Zaharia
      2. MAPREDUCE-2505.patch
        2 kB
        Matei Zaharia

        Activity

        Hide
        Matei Zaharia added a comment -

        Here's a patch to add this documentation.

        Show
        Matei Zaharia added a comment - Here's a patch to add this documentation.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12479517/MAPREDUCE-2505.patch
        against trunk revision 1103993.

        +1 @author. The patch does not contain any @author tags.

        +0 tests included. The patch appears to be a documentation patch that doesn't require tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/261//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/261//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/261//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12479517/MAPREDUCE-2505.patch against trunk revision 1103993. +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/261//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/261//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/261//console This message is automatically generated.
        Hide
        Eli Collins added a comment -

        +1 lgtm

        Show
        Eli Collins added a comment - +1 lgtm
        Hide
        Eli Collins added a comment -

        I've committed this to branch 22 and trunk. Thanks Matei!

        Show
        Eli Collins added a comment - I've committed this to branch 22 and trunk. Thanks Matei!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #681 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/681/)
        MAPREDUCE-2505. Explain how to use ACLs in the fair scheduler. Contributed by Matei Zaharia.

        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1124374
        Files :

        • /hadoop/mapreduce/trunk/src/docs/src/documentation/content/xdocs/fair_scheduler.xml
        • /hadoop/mapreduce/trunk/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #681 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/681/ ) MAPREDUCE-2505 . Explain how to use ACLs in the fair scheduler. Contributed by Matei Zaharia. eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1124374 Files : /hadoop/mapreduce/trunk/src/docs/src/documentation/content/xdocs/fair_scheduler.xml /hadoop/mapreduce/trunk/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-22-branch #54 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/54/)
        MAPREDUCE-2505. svn merge -c 1124374 from trunk

        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1124377
        Files :

        • /hadoop/mapreduce/branches/branch-0.22/conf
        • /hadoop/mapreduce/branches/branch-0.22/src/contrib/data_join
        • /hadoop/mapreduce/branches/branch-0.22/src/test/mapred/org/apache/hadoop/hdfs
        • /hadoop/mapreduce/branches/branch-0.22/src/test/mapred/org/apache/hadoop/ipc
        • /hadoop/mapreduce/branches/branch-0.22/src/contrib/streaming
        • /hadoop/mapreduce/branches/branch-0.22/src/contrib/build-contrib.xml
        • /hadoop/mapreduce/branches/branch-0.22/src/contrib/capacity-scheduler
        • /hadoop/mapreduce/branches/branch-0.22/src/test/mapred/org/apache/hadoop/fs
        • /hadoop/mapreduce/branches/branch-0.22/src/docs/src/documentation/content/xdocs/fair_scheduler.xml
        • /hadoop/mapreduce/branches/branch-0.22/src/contrib/mrunit
        • /hadoop/mapreduce/branches/branch-0.22/CHANGES.txt
        • /hadoop/mapreduce/branches/branch-0.22/src/contrib/block_forensics
        • /hadoop/mapreduce/branches/branch-0.22/conf/capacity-scheduler.xml.template
        • /hadoop/mapreduce/branches/branch-0.22/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java
        • /hadoop/mapreduce/branches/branch-0.22/src/java
        • /hadoop/mapreduce/branches/branch-0.22/src/test/mapred
        • /hadoop/mapreduce/branches/branch-0.22/src/webapps/job
        • /hadoop/mapreduce/branches/branch-0.22/src/contrib/vaidya
        • /hadoop/mapreduce/branches/branch-0.22/src/test/mapred/org/apache/hadoop/io/FileBench.java
        • /hadoop/mapreduce/branches/branch-0.22/src/contrib/dynamic-scheduler
        • /hadoop/mapreduce/branches/branch-0.22/src/contrib
        • /hadoop/mapreduce/branches/branch-0.22/src/examples
        • /hadoop/mapreduce/branches/branch-0.22
        • /hadoop/mapreduce/branches/branch-0.22/.gitignore
        • /hadoop/mapreduce/branches/branch-0.22/src/contrib/index
        • /hadoop/mapreduce/branches/branch-0.22/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java
        • /hadoop/mapreduce/branches/branch-0.22/src/contrib/eclipse-plugin
        • /hadoop/mapreduce/branches/branch-0.22/src/contrib/build.xml
        • /hadoop/mapreduce/branches/branch-0.22/src/contrib/fairscheduler
        • /hadoop/mapreduce/branches/branch-0.22/src/c++
        • /hadoop/mapreduce/branches/branch-0.22/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #54 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-22-branch/54/ ) MAPREDUCE-2505 . svn merge -c 1124374 from trunk eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1124377 Files : /hadoop/mapreduce/branches/branch-0.22/conf /hadoop/mapreduce/branches/branch-0.22/src/contrib/data_join /hadoop/mapreduce/branches/branch-0.22/src/test/mapred/org/apache/hadoop/hdfs /hadoop/mapreduce/branches/branch-0.22/src/test/mapred/org/apache/hadoop/ipc /hadoop/mapreduce/branches/branch-0.22/src/contrib/streaming /hadoop/mapreduce/branches/branch-0.22/src/contrib/build-contrib.xml /hadoop/mapreduce/branches/branch-0.22/src/contrib/capacity-scheduler /hadoop/mapreduce/branches/branch-0.22/src/test/mapred/org/apache/hadoop/fs /hadoop/mapreduce/branches/branch-0.22/src/docs/src/documentation/content/xdocs/fair_scheduler.xml /hadoop/mapreduce/branches/branch-0.22/src/contrib/mrunit /hadoop/mapreduce/branches/branch-0.22/CHANGES.txt /hadoop/mapreduce/branches/branch-0.22/src/contrib/block_forensics /hadoop/mapreduce/branches/branch-0.22/conf/capacity-scheduler.xml.template /hadoop/mapreduce/branches/branch-0.22/src/test/mapred/org/apache/hadoop/io/TestSequenceFileMergeProgress.java /hadoop/mapreduce/branches/branch-0.22/src/java /hadoop/mapreduce/branches/branch-0.22/src/test/mapred /hadoop/mapreduce/branches/branch-0.22/src/webapps/job /hadoop/mapreduce/branches/branch-0.22/src/contrib/vaidya /hadoop/mapreduce/branches/branch-0.22/src/test/mapred/org/apache/hadoop/io/FileBench.java /hadoop/mapreduce/branches/branch-0.22/src/contrib/dynamic-scheduler /hadoop/mapreduce/branches/branch-0.22/src/contrib /hadoop/mapreduce/branches/branch-0.22/src/examples /hadoop/mapreduce/branches/branch-0.22 /hadoop/mapreduce/branches/branch-0.22/.gitignore /hadoop/mapreduce/branches/branch-0.22/src/contrib/index /hadoop/mapreduce/branches/branch-0.22/src/test/mapred/org/apache/hadoop/security/authorize/TestServiceLevelAuthorization.java /hadoop/mapreduce/branches/branch-0.22/src/contrib/eclipse-plugin /hadoop/mapreduce/branches/branch-0.22/src/contrib/build.xml /hadoop/mapreduce/branches/branch-0.22/src/contrib/fairscheduler /hadoop/mapreduce/branches/branch-0.22/src/c++ /hadoop/mapreduce/branches/branch-0.22/src/test/mapred/org/apache/hadoop/test/MapredTestDriver.java
        Hide
        Matei Zaharia added a comment -

        Thanks for committing this, Eli. I'm going to commit it to the 0.20 and 0.21 branches as well, with a minor modification (the queue name property is called something different in 0.20).

        Show
        Matei Zaharia added a comment - Thanks for committing this, Eli. I'm going to commit it to the 0.20 and 0.21 branches as well, with a minor modification (the queue name property is called something different in 0.20).
        Hide
        Eli Collins added a comment -

        My pleasure. Should be mapred.job.queue.name in branch 20.

        Show
        Eli Collins added a comment - My pleasure. Should be mapred.job.queue.name in branch 20.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #684 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/684/)
        MAPREDUCE-2505. Explain how to use ACLs in the fair scheduler. Contributed by Matei Zaharia.

        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1124374
        Files :

        • /hadoop/mapreduce/trunk/src/docs/src/documentation/content/xdocs/fair_scheduler.xml
        • /hadoop/mapreduce/trunk/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #684 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk/684/ ) MAPREDUCE-2505 . Explain how to use ACLs in the fair scheduler. Contributed by Matei Zaharia. eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1124374 Files : /hadoop/mapreduce/trunk/src/docs/src/documentation/content/xdocs/fair_scheduler.xml /hadoop/mapreduce/trunk/CHANGES.txt
        Hide
        Matei Zaharia added a comment -

        Here's a patch for 0.20.203, but I'm a bit confused about where the work is being done for the next release. Should I put this in the 0.20-security branch, or 0.20.203, or 0.20.204?

        BTW I've decided to only commit this to the security branches because a) few people are likely to use new releases of 0.20.x and 0.21 and b) the security branches actually have security for the ACLs.

        Show
        Matei Zaharia added a comment - Here's a patch for 0.20.203, but I'm a bit confused about where the work is being done for the next release. Should I put this in the 0.20-security branch, or 0.20.203, or 0.20.204? BTW I've decided to only commit this to the security branches because a) few people are likely to use new releases of 0.20.x and 0.21 and b) the security branches actually have security for the ACLs.

          People

          • Assignee:
            Matei Zaharia
            Reporter:
            Matei Zaharia
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development