Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: security
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      * Removed aclsEnabled flag from queues configuration files.
      * Removed the configuration property mapreduce.cluster.job-authorization-enabled.
      * Added mapreduce.cluster.acls.enabled as the single configuration property in mapred-default.xml that enables the authorization checks for all job level and queue level operations.
      * To enable authorization of users to do job level and queue level operations, mapreduce.cluster.acls.enabled is to be set to true in JobTracker's configuration and in all TaskTrackers' configurations.
      * To get access to a job, it is enough for a user to be part of one of the access lists i.e. either job-acl or queue-admins-acl(unlike before, when, one has to be part of both the lists).
      * Queue administrators(configured via acl-administer-jobs) of a queue can do all view-job and modify-job operations on all jobs submitted to that queue.
      * ClusterOwner(who started the mapreduce cluster) and cluster administrators(configured via mapreduce.cluster.permissions.supergroup) can do all job level operations and queue level operations on all jobs on all queues in that cluster irrespective of job-acls and queue-acls configured.
      * JobOwner(who submitted job to a queue) can do all view-job and modify-job operations on his/her job irrespective of job-acls and queue-acls.
      * Since aclsEnabled flag is removed from queues configuration files, "refresh of queues configuration" will not change mapreduce.cluster.acls.enabled on the fly. mapreduce.cluster.acls.enabled can be modified only when restarting the mapreduce cluster.
      Show
      * Removed aclsEnabled flag from queues configuration files. * Removed the configuration property mapreduce.cluster.job-authorization-enabled. * Added mapreduce.cluster.acls.enabled as the single configuration property in mapred-default.xml that enables the authorization checks for all job level and queue level operations. * To enable authorization of users to do job level and queue level operations, mapreduce.cluster.acls.enabled is to be set to true in JobTracker's configuration and in all TaskTrackers' configurations. * To get access to a job, it is enough for a user to be part of one of the access lists i.e. either job-acl or queue-admins-acl(unlike before, when, one has to be part of both the lists). * Queue administrators(configured via acl-administer-jobs) of a queue can do all view-job and modify-job operations on all jobs submitted to that queue. * ClusterOwner(who started the mapreduce cluster) and cluster administrators(configured via mapreduce.cluster.permissions.supergroup) can do all job level operations and queue level operations on all jobs on all queues in that cluster irrespective of job-acls and queue-acls configured. * JobOwner(who submitted job to a queue) can do all view-job and modify-job operations on his/her job irrespective of job-acls and queue-acls. * Since aclsEnabled flag is removed from queues configuration files, "refresh of queues configuration" will not change mapreduce.cluster.acls.enabled on the fly. mapreduce.cluster.acls.enabled can be modified only when restarting the mapreduce cluster.

      Description

      MAPREDUCE-1307 introduced job ACLs for securing job level operations. So in current trunk, queue ACLs and job ACLs are checked(with AND for both acls) for allowing job level operations. So for doing operations like killJob, killTask and setJobPriority user should be part of both mapred.queue.

      {queuename}.acl-administer-jobs and in mapreduce.job.acl-modify-job. This needs to change so that users who are part of mapred.queue.{queuename}

      .acl-administer-jobs will be able to do killJob,killTask,setJobPriority and users part of mapreduce.job.acl-modify-job will be able to do killJob,killTask,setJobPriority.

      1. 1664.20S.3.4.patch
        123 kB
        Ravi Gummadi
      2. 1664.patch
        194 kB
        Ravi Gummadi
      3. 1664.qAdminsJobView.20S.v1.6.patch
        97 kB
        Ravi Gummadi
      4. 1664.v1.1.patch
        212 kB
        Ravi Gummadi
      5. 1664.v1.2.patch
        225 kB
        Ravi Gummadi
      6. 1664.v1.patch
        210 kB
        Ravi Gummadi
      7. M1664y20s-testfix.patch
        9 kB
        Chris Douglas
      8. mr-1664-20-bugfix.patch
        2 kB
        Devaraj Das

        Issue Links

          Activity

          Hide
          Hudson added a comment -

          Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/)

          Show
          Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #523 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/523/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk-Commit #378 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/378/)

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #378 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk-Commit/378/ )
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Common-trunk #453 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/453/)
          HADOOP-6922. COMMON part of MAPREDUCE-1664. Makes AccessControlList a writable and updates documentation for Job ACLs. Contributed by Ravi Gummadi.

          Show
          Hudson added a comment - Integrated in Hadoop-Common-trunk #453 (See https://hudson.apache.org/hudson/job/Hadoop-Common-trunk/453/ ) HADOOP-6922 . COMMON part of MAPREDUCE-1664 . Makes AccessControlList a writable and updates documentation for Job ACLs. Contributed by Ravi Gummadi.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          I just committed this to trunk. Thanks Ravi!

          Show
          Vinod Kumar Vavilapalli added a comment - I just committed this to trunk. Thanks Ravi!
          Hide
          Ravi Gummadi added a comment -

          ant test passed on my local machine.

          ant test-patch gave:

          [exec] +1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 63 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          [exec]
          [exec] +1 system tests framework. The patch passed system tests framework compile.

          On a single node cluster, tested different cases for job acls and queue acls and supergroup and cluster-owner(user who started the cluster) and job-owner. Everything seem to be working as expected.

          Show
          Ravi Gummadi added a comment - ant test passed on my local machine. ant test-patch gave: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 63 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system tests framework. The patch passed system tests framework compile. On a single node cluster, tested different cases for job acls and queue acls and supergroup and cluster-owner(user who started the cluster) and job-owner. Everything seem to be working as expected.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          +1 for the patch. Please post the test-patch and test results so I can commit it together with HADOOP-6922.

          Show
          Vinod Kumar Vavilapalli added a comment - +1 for the patch. Please post the test-patch and test results so I can commit it together with HADOOP-6922 .
          Hide
          Ravi Gummadi added a comment -

          Attaching new patch with minor changes and to make the patch not dependent on the patch of MAPREDUCE-1550 as MAPREDUCE-1550 is resolved as invalid for trunk.

          Show
          Ravi Gummadi added a comment - Attaching new patch with minor changes and to make the patch not dependent on the patch of MAPREDUCE-1550 as MAPREDUCE-1550 is resolved as invalid for trunk.
          Hide
          Ravi Gummadi added a comment -

          Attaching new patch incorporating review comments.

          This patch is on top of patch for MAPREDUCE-1550.

          Show
          Ravi Gummadi added a comment - Attaching new patch incorporating review comments. This patch is on top of patch for MAPREDUCE-1550 .
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Minor comments:

          • mapred-default.xml: "For enabling this flag, this is to be set to true on JobTracker's configuration file and in all TaskTracker's configuration files." can better be "For enabling this flag, this is to be set to true on the configuration files of JobTracker and all the TaskTrackers"
          • QueueConfigurationParser: aclsEnabled isn't really deprecated as we ignore it completely. May be we should simply say that in the message.
          • QueueInfo.toFullPropertyName(): QueueInfo is public API. Let's not put this method in there. You can let it be in QueueManager by making QueueManager public but with private visibility classification. Java public scope identifier doesn't really promise the public nature of a class/api anymore at all. You should definitely also put java comments as to where all this method can be used.
          • ClusterMapReduceTestCase.startCluster() with the added parameter argument seems like a very weird API to have, it isn't even generic enough to accept other information. We can just have this in TestWebUIAuthorization.
          • QueueManagerTestUtils.createQueuesConfigFile() doesn't need to return a configuration. Adding the mapred-queues.xml as a resource to a configuration object is also useless.
          • TestJobHistory.testJobHistory(). Do we need a call to createQueuesConfigFile here? It is configuring default queue again, yet the file isn't being put in the classpath. Same in TestRecoveryManager.
          • TestQueueAclsForCurrentUser.setupConfForNoAccess() : The admin acls configured are different now. Earlier they were u1 for qu1 and " g2" for qu2.
          Show
          Vinod Kumar Vavilapalli added a comment - Minor comments: mapred-default.xml: "For enabling this flag, this is to be set to true on JobTracker's configuration file and in all TaskTracker's configuration files." can better be "For enabling this flag, this is to be set to true on the configuration files of JobTracker and all the TaskTrackers" QueueConfigurationParser: aclsEnabled isn't really deprecated as we ignore it completely. May be we should simply say that in the message. QueueInfo.toFullPropertyName(): QueueInfo is public API. Let's not put this method in there. You can let it be in QueueManager by making QueueManager public but with private visibility classification. Java public scope identifier doesn't really promise the public nature of a class/api anymore at all. You should definitely also put java comments as to where all this method can be used. ClusterMapReduceTestCase.startCluster() with the added parameter argument seems like a very weird API to have, it isn't even generic enough to accept other information. We can just have this in TestWebUIAuthorization. QueueManagerTestUtils.createQueuesConfigFile() doesn't need to return a configuration. Adding the mapred-queues.xml as a resource to a configuration object is also useless. TestJobHistory.testJobHistory(). Do we need a call to createQueuesConfigFile here? It is configuring default queue again, yet the file isn't being put in the classpath. Same in TestRecoveryManager. TestQueueAclsForCurrentUser.setupConfForNoAccess() : The admin acls configured are different now. Earlier they were u1 for qu1 and " g2" for qu2.
          Hide
          Ravi Gummadi added a comment -

          Attaching new patch incorporating review comments.

          Show
          Ravi Gummadi added a comment - Attaching new patch incorporating review comments.
          Hide
          Ravi Gummadi added a comment -

          >> JobInProgress.java: you accidentally deleted a line if (jobname == null)

          { jobname = ""; }

          +720 line after applying patch and +727 before

          This is intentionally removed because jobName cannot be null here.

          >> JobSubmittedEvent: Please don't mess with deprecated constructors. Create new ones as you wish. Once you do this, the changes in rumen/20history parser will not be needed at all.

          As JobSubmittedEvent is flagged as @InterfaceAudience.Private, am fully removing the unnecessary constructors and keeping the only (modified)one that is needed.

          >> TestTaskTrackerLocalization: we were earlier checking permissions of job-acls file. The file got moved into job-log-dir, but the permissions checks are missed now.

          This was not missed. File permissions are validated first and then file content.

          Incorporating other review comments.

          Show
          Ravi Gummadi added a comment - >> JobInProgress.java: you accidentally deleted a line if (jobname == null) { jobname = ""; } +720 line after applying patch and +727 before This is intentionally removed because jobName cannot be null here. >> JobSubmittedEvent: Please don't mess with deprecated constructors. Create new ones as you wish. Once you do this, the changes in rumen/20history parser will not be needed at all. As JobSubmittedEvent is flagged as @InterfaceAudience.Private, am fully removing the unnecessary constructors and keeping the only (modified)one that is needed. >> TestTaskTrackerLocalization: we were earlier checking permissions of job-acls file. The file got moved into job-log-dir, but the permissions checks are missed now. This was not missed. File permissions are validated first and then file content. Incorporating other review comments.
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Started looking at the patch. It's big. So I am summarizing user facing changes to be explicit.

          User facing changes

          • The main change is that, now, access to view or modify a job is controlled by job-acls and queue-acls. To get access to a job, it is enough for a user to be part of one of the access lists. (unlike before, when, one has to be part of both the lists).
          • job-authorization configuration boolean flag is removed.
          • both job and queue acls are now driven by "mapred.acls.enabled"
          • mapred.acls.enabled is now also needed in TTs conf. (0.20 and pre-0.20 it was only needed on JT)
          • In 21, admins could do refresh queues and switch off and on the queue-acls feature. This isn't going to happen anymore. Note that this on and off feature wasn't there in 0.20, and even in 0.21, I can hazard-guess and say it is an 'accidental' feature.

          --------------------------------------------------------------------------------------------------------------------------------------

          Code-review comments:

          • mapred-default.xml: Fix documentation for mapred.acls.enabled to mention that it is now needed in both JT's and TT's conf files, if at all.
          • Deprecate mapred.acls.enabled (fix the documentation too) in favour of mapreduce.acls.enabled
          • aclsEnabled should still be read for the sake of spitting out deprecation warnings?
          • I think its more natural to still keep QueueManager.getQueueConfigurationParser() as a static factory, that was its original intention. You can simply pass in the acls enabled flag also as an argument to the factory method.
          • Once you do the above, QueueManager.dumpConfiguration also can remain static, with no additional arguments.
          • JobInProgress.java: you accidentally deleted a line if (jobname == null) { jobname = ""; }

            +720 line after applying patch and +727 before

          • JobSubmittedEvent: Please don't mess with deprecated constructors. Create new ones as you wish. Once you do this, the changes in rumen/20history parser will not be needed at all.
          • JobSubmitter.java: Hard coded internal acl config names are bad. Please use a constant/method/utility but mark it internal only.
          • We should really not bring back the old constant DeprecatedQueueConfigurationParser.MAPRED_QUEUE_NAMES_KEY and the corresponding deprecated conf name "mapred.queue.names". Please replace all the uses of this with the new config file and properties (except possibly when testing the deprecated configuration)
          • JobTracker.java: You can remove AuditLogger.logSuccess call in addJob(). This will be duplicated by the one in aclsmanager.

          Tests:

          • TestQueueManagerWithJobTracker.testAclsDisabled(), you can remove the refreshing of queues operation.. it doesn't add any value now.
          • TestTaskTrackerLocalization: we were earlier checking permissions of job-acls file. The file got moved into job-log-dir, but the permissions checks are missed now.
          Show
          Vinod Kumar Vavilapalli added a comment - Started looking at the patch. It's big. So I am summarizing user facing changes to be explicit. User facing changes The main change is that, now, access to view or modify a job is controlled by job-acls and queue-acls. To get access to a job, it is enough for a user to be part of one of the access lists. (unlike before, when, one has to be part of both the lists). job-authorization configuration boolean flag is removed. both job and queue acls are now driven by "mapred.acls.enabled" mapred.acls.enabled is now also needed in TTs conf. (0.20 and pre-0.20 it was only needed on JT) In 21, admins could do refresh queues and switch off and on the queue-acls feature. This isn't going to happen anymore. Note that this on and off feature wasn't there in 0.20, and even in 0.21, I can hazard-guess and say it is an 'accidental' feature. -------------------------------------------------------------------------------------------------------------------------------------- Code-review comments: mapred-default.xml: Fix documentation for mapred.acls.enabled to mention that it is now needed in both JT's and TT's conf files, if at all. Deprecate mapred.acls.enabled (fix the documentation too) in favour of mapreduce.acls.enabled aclsEnabled should still be read for the sake of spitting out deprecation warnings? I think its more natural to still keep QueueManager.getQueueConfigurationParser() as a static factory, that was its original intention. You can simply pass in the acls enabled flag also as an argument to the factory method. Once you do the above, QueueManager.dumpConfiguration also can remain static, with no additional arguments. JobInProgress.java: you accidentally deleted a line if (jobname == null) { jobname = ""; } +720 line after applying patch and +727 before JobSubmittedEvent: Please don't mess with deprecated constructors. Create new ones as you wish. Once you do this, the changes in rumen/20history parser will not be needed at all. JobSubmitter.java: Hard coded internal acl config names are bad. Please use a constant/method/utility but mark it internal only. We should really not bring back the old constant DeprecatedQueueConfigurationParser.MAPRED_QUEUE_NAMES_KEY and the corresponding deprecated conf name "mapred.queue.names". Please replace all the uses of this with the new config file and properties (except possibly when testing the deprecated configuration) JobTracker.java: You can remove AuditLogger.logSuccess call in addJob(). This will be duplicated by the one in aclsmanager. Tests: TestQueueManagerWithJobTracker.testAclsDisabled(), you can remove the refreshing of queues operation.. it doesn't add any value now. TestTaskTrackerLocalization: we were earlier checking permissions of job-acls file. The file got moved into job-log-dir, but the permissions checks are missed now.
          Hide
          Ravi Gummadi added a comment -

          Attaching patch for trunk.

          Unit tests passed on my local machine. Will do some web UI based testing with a single node cluster on my local machine.
          Meanwhile, please review and provide your comments.

          Show
          Ravi Gummadi added a comment - Attaching patch for trunk. Unit tests passed on my local machine. Will do some web UI based testing with a single node cluster on my local machine. Meanwhile, please review and provide your comments.
          Hide
          Ravi Gummadi added a comment -

          For changes to AccessControlList.java mentioned in (13) in earlier comment and for the documentation changes to cluster_setup.xml, a new JIRA in COMMON project is created. It is HADOOP-6922. Attached patch for HADOOP-6922.

          Show
          Ravi Gummadi added a comment - For changes to AccessControlList.java mentioned in (13) in earlier comment and for the documentation changes to cluster_setup.xml, a new JIRA in COMMON project is created. It is HADOOP-6922 . Attached patch for HADOOP-6922 .
          Hide
          Ravi Gummadi added a comment -

          Working on the forward port(to trunk) of the combined patch of the 4 patches attached earlier.

          Some of the new issues for this patch in trunk are
          (1) mapred.acls.enabled is removed from mapred-default.xml and moved to mapred-queues.xml.
          (2) Configuration of queue acls is moved to mapred-queues.xml.

          In this forward port patch, I am adding the config property mapred.acls.enabled back into mapred-default.xml and this drives the need for authorization of both queue level operations and job level operations.

          Show
          Ravi Gummadi added a comment - Working on the forward port(to trunk) of the combined patch of the 4 patches attached earlier. Some of the new issues for this patch in trunk are (1) mapred.acls.enabled is removed from mapred-default.xml and moved to mapred-queues.xml. (2) Configuration of queue acls is moved to mapred-queues.xml. In this forward port patch, I am adding the config property mapred.acls.enabled back into mapred-default.xml and this drives the need for authorization of both queue level operations and job level operations.
          Hide
          Ravi Gummadi added a comment -

          Already attached 4 patches form the combined patch for Yahoo security branch 20.1xx.

          At a higher level, the above 4 patches did the following:

          (1) Removed the flag mapreduce.cluster.job-authorization-enabled and let the flag mapred.acls.enabled drive both queue acls checks and job acls checks. All users will be able to do queue level and job level operations if mapred.acls.enabled is set to false. If mapred.acls.enabled is set to true, then the UNION of users part of queue ACLs and job ACLs will be able to do operation if that operation is driven by both queue ACLs and job ACLs — like killJob, killTask and setJobPriority.

          (2) JobHistory is changed to have "*" as value for the job acls' keys VIEW_JOB and MODIFY_JOB in the line of job submission in history file when mapred.acls.enabled is false. This would allow all users to view job-history of this job even when we restart the cluster with mapred.acls.enabled set to true.

          (3) JobHistory is changed to have job queue name in the job submission line in history file. This is used to allow queue admins of that queue to view history details when serving job history based queries.

          (4) Created new enum "Operation" that contains the actual operations like SUBMIT_JOB, KILL_TASK, FAIL_TASK, KILL_JOB,SET_JOB_PRIORITY, VIEW_TASK_LOGS, VIEW_JOB_DETAILS, VIEW_JOB_COUNTERS.

          (5) Added new class ACLsManager that is responsible for authorization of all queue level and job level operations.
          ACLsManager internally maintains MROwner(user who started mapreduce cluster), cluster administrators(configured by MROwner via mapreduce.cluster.permissions.supergroup). ACLsManager uses QueueManager and JobACLsManager for checking queue acls and job acls respectively for authorizing users to do various operations.

          (6) Operation SUBMIT_JOB is solely driven by the queue acl acl-submit-job. The operations VIEW_JOB_DETAILS, VIEW_JOB_COUNTERS, VIEW_TASK_LOGS, FAIL_TASK, KILL_TASK, KILL_JOB, SET_JOB_PRIORITY are driven by both job acls(either acl-view-job OR acl-modify-job based on the operation) and queue acl acl-administer-jobs. If a user is part of job acl that drives the specific operation OR part of the queue acl acl-administer-jobs, then he/she is an authorized user to perform the operation. Irrespective of queue acls and job acls configured, any operation can be done by MROwner and cluster administrators(members of supergroup).

          (7) Default values for the queue acls is changed from "*"(all users allowed by default) to " "(no user is allowed by default). This is to make the default values of queue acls consistent with job acls.

          (8) Since TaskTracker will not have job acls once the job finishes execution, the job acls are persisted to userlogs/$jobid/job-acls.xml so that single file per job is maintained. This file contains job's acl-view-job, job queue name, job owner name, queue admins acl of the queue to which this job was submitted to. These info will be used by TaskTracker to do authorization of users for accessing task logs(in TaskLogServlet).

          (9) Made LinuxTaskController to set permissions for job acls file also when permissions are being set for job userlog dir.

          (10) For TaskTrackers to write queue admins of specific job queue to job-acls.xml file, the queue admins acl is written to job conf by job client. Made JobClient to get queue admins(of the queue to which the job is being submitted to) from JobTracker and set this ACL in job conf. Tasktracker reads this queue admins ACL from job conf and writes to job-acls.xml file at the time of job localization.

          (11) TaskLogServlet checks the queue admins ACL explicitly(as there is no Queuemanager on TaskTracker) before checking for other things like jobOwner, cluster admins and view-job-acl. The queue admins ACL and view-job-acl are obtained from job-acls.xml file.

          (12) Changed readCounters() in CompletedJobStatusStore so that queue name is obtained from job profile and thus allowing job-view of that job for the queue admins of that queue.

          (13) Since the newly added method getQueueAdmins() in ClientProtocol returns AccessControlList object directly, creation of AccessControlList object through Reflection utils is needed. This needed a new constructor for AccessControlList and that to be registered in WritableFactories.

          Show
          Ravi Gummadi added a comment - Already attached 4 patches form the combined patch for Yahoo security branch 20.1xx. At a higher level, the above 4 patches did the following: (1) Removed the flag mapreduce.cluster.job-authorization-enabled and let the flag mapred.acls.enabled drive both queue acls checks and job acls checks. All users will be able to do queue level and job level operations if mapred.acls.enabled is set to false. If mapred.acls.enabled is set to true, then the UNION of users part of queue ACLs and job ACLs will be able to do operation if that operation is driven by both queue ACLs and job ACLs — like killJob, killTask and setJobPriority. (2) JobHistory is changed to have "*" as value for the job acls' keys VIEW_JOB and MODIFY_JOB in the line of job submission in history file when mapred.acls.enabled is false. This would allow all users to view job-history of this job even when we restart the cluster with mapred.acls.enabled set to true. (3) JobHistory is changed to have job queue name in the job submission line in history file. This is used to allow queue admins of that queue to view history details when serving job history based queries. (4) Created new enum "Operation" that contains the actual operations like SUBMIT_JOB, KILL_TASK, FAIL_TASK, KILL_JOB,SET_JOB_PRIORITY, VIEW_TASK_LOGS, VIEW_JOB_DETAILS, VIEW_JOB_COUNTERS. (5) Added new class ACLsManager that is responsible for authorization of all queue level and job level operations. ACLsManager internally maintains MROwner(user who started mapreduce cluster), cluster administrators(configured by MROwner via mapreduce.cluster.permissions.supergroup). ACLsManager uses QueueManager and JobACLsManager for checking queue acls and job acls respectively for authorizing users to do various operations. (6) Operation SUBMIT_JOB is solely driven by the queue acl acl-submit-job. The operations VIEW_JOB_DETAILS, VIEW_JOB_COUNTERS, VIEW_TASK_LOGS, FAIL_TASK, KILL_TASK, KILL_JOB, SET_JOB_PRIORITY are driven by both job acls(either acl-view-job OR acl-modify-job based on the operation) and queue acl acl-administer-jobs. If a user is part of job acl that drives the specific operation OR part of the queue acl acl-administer-jobs, then he/she is an authorized user to perform the operation. Irrespective of queue acls and job acls configured, any operation can be done by MROwner and cluster administrators(members of supergroup). (7) Default values for the queue acls is changed from "*"(all users allowed by default) to " "(no user is allowed by default). This is to make the default values of queue acls consistent with job acls. (8) Since TaskTracker will not have job acls once the job finishes execution, the job acls are persisted to userlogs/$jobid/job-acls.xml so that single file per job is maintained. This file contains job's acl-view-job, job queue name, job owner name, queue admins acl of the queue to which this job was submitted to. These info will be used by TaskTracker to do authorization of users for accessing task logs(in TaskLogServlet). (9) Made LinuxTaskController to set permissions for job acls file also when permissions are being set for job userlog dir. (10) For TaskTrackers to write queue admins of specific job queue to job-acls.xml file, the queue admins acl is written to job conf by job client. Made JobClient to get queue admins(of the queue to which the job is being submitted to) from JobTracker and set this ACL in job conf. Tasktracker reads this queue admins ACL from job conf and writes to job-acls.xml file at the time of job localization. (11) TaskLogServlet checks the queue admins ACL explicitly(as there is no Queuemanager on TaskTracker) before checking for other things like jobOwner, cluster admins and view-job-acl. The queue admins ACL and view-job-acl are obtained from job-acls.xml file. (12) Changed readCounters() in CompletedJobStatusStore so that queue name is obtained from job profile and thus allowing job-view of that job for the queue admins of that queue. (13) Since the newly added method getQueueAdmins() in ClientProtocol returns AccessControlList object directly, creation of AccessControlList object through Reflection utils is needed. This needed a new constructor for AccessControlList and that to be registered in WritableFactories.
          Hide
          Ravi Gummadi added a comment -

          I haven't started on the forward port. It would involve porting of all the 4 patches uploaded to this JIRA. Would need some discussion here on the design/implementation decisions before finalizing the design for trunk. Feel free to assign this JIRA to you and work on this if you have time.

          Show
          Ravi Gummadi added a comment - I haven't started on the forward port. It would involve porting of all the 4 patches uploaded to this JIRA. Would need some discussion here on the design/implementation decisions before finalizing the design for trunk. Feel free to assign this JIRA to you and work on this if you have time.
          Hide
          Krishna Ramachandran added a comment -

          Ravi
          where are we on this now (also the forward port)? I finished mapreduce-1759 and requires 1664

          Show
          Krishna Ramachandran added a comment - Ravi where are we on this now (also the forward port)? I finished mapreduce-1759 and requires 1664
          Hide
          Ravi Gummadi added a comment -

          Access to task logs is given to the administrators of the queue only as they should have access to everything on the jobs of the queue. No ?
          So the users part of the queue ACL acl-administer-jobs will be able to "view job details" in addition to the allowed operations SET_JOB_PRIORITY, KILL_TASK, KILL_JOB.

          Show
          Ravi Gummadi added a comment - Access to task logs is given to the administrators of the queue only as they should have access to everything on the jobs of the queue. No ? So the users part of the queue ACL acl-administer-jobs will be able to "view job details" in addition to the allowed operations SET_JOB_PRIORITY, KILL_TASK, KILL_JOB.
          Hide
          Allen Wittenauer added a comment -

          Isn't getting access to task logs a security risk?

          Show
          Allen Wittenauer added a comment - Isn't getting access to task logs a security risk?
          Hide
          Ravi Gummadi added a comment -

          Attaching patch for earlier version of hadoop, not for commit here. This patch is on top of previous patches. This patch enables queue administrators of a queue to view job details(job counters, task logs, etc) of jobs submitted to that queue even though they are not part of acl-view-job.

          Show
          Ravi Gummadi added a comment - Attaching patch for earlier version of hadoop, not for commit here. This patch is on top of previous patches. This patch enables queue administrators of a queue to view job details(job counters, task logs, etc) of jobs submitted to that queue even though they are not part of acl-view-job.
          Hide
          Devaraj Das added a comment -

          Bugfix on the top of the previous patch. Not for commit.

          Show
          Devaraj Das added a comment - Bugfix on the top of the previous patch. Not for commit.
          Hide
          Chris Douglas added a comment -

          Attaching fix to testcases on Ravi's behalf

          Show
          Chris Douglas added a comment - Attaching fix to testcases on Ravi's behalf
          Hide
          Ravi Gummadi added a comment -

          Attaching patch for earlier version of hadoop. Not for commit here.

          Show
          Ravi Gummadi added a comment - Attaching patch for earlier version of hadoop. Not for commit here.

            People

            • Assignee:
              Ravi Gummadi
              Reporter:
              Ravi Gummadi
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development