Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-5132

Enhance FairScheduler queue selection with Unix-SubGroups

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I have clients that use the primary group for other reasons and would like to select the fairScheduler queue by subgroup.

      Currently the fairscheduler can only link a job config property to a fairscheduler queue and subgroups are not in the job config.

      Now thankfully we can get this data from UserGroupInformation.createProxyUser which is cool because this is how TT currently see if it can copy data when kerberos is enabled. Once we have the sub-groups we just connect the first sub-group that matches a queue.

      I'll put up some code that I used to solve the problem in the short term.

      Note that there is an issue with this code. It doesn't populate the TT Web UI drop down menu correctly. I think I have a solution for that, but before I implement that I was wondering if the community has any feedback first.

      1. MAPREDUCE-5132.NotTested.patch
        9 kB
        Ted Malaska
      2. FairSchedulerPlusCode.zip
        5 kB
        Ted Malaska

        Activity

        Hide
        Sandy Ryza added a comment -

        Definitely seems like a useful feature. This is likely to be the first of many policies that automatically map submitted jobs to queues. It's possible that we would want to choose a queue based on the amount of resources a job is asking for, or even perhaps the name of the mapper class. Before handling this case in particular, I think we should have a more unified approach for where we want this job classification to happen. As the resource manager will only have access to a limited subset of information about a job, I'm wondering whether it might make more sense to implement on the client side?

        Show
        Sandy Ryza added a comment - Definitely seems like a useful feature. This is likely to be the first of many policies that automatically map submitted jobs to queues. It's possible that we would want to choose a queue based on the amount of resources a job is asking for, or even perhaps the name of the mapper class. Before handling this case in particular, I think we should have a more unified approach for where we want this job classification to happen. As the resource manager will only have access to a limited subset of information about a job, I'm wondering whether it might make more sense to implement on the client side?
        Hide
        Ted Malaska added a comment -

        This is a untested implement of how a users groups could be used to select a fair scheduler queue.

        Show
        Ted Malaska added a comment - This is a untested implement of how a users groups could be used to select a fair scheduler queue.
        Hide
        Ted Malaska added a comment -

        Just had a great talk with Karthik Kambatla and we have a starting design and approach.

        Design: We are going to make Sub-Groups work like user-name does today. Because user-name doesn't pull from the job config because of kerberos and security issues.

        Approach:
        1. I am going to assign the JIRI to my self.
        2. I am going to implement a truck (MR2-YERN) solution first. Then back port that solution to branch-1 (MR1)
        3. I will try to get the first patch done by end of week next week.

        Thx

        Show
        Ted Malaska added a comment - Just had a great talk with Karthik Kambatla and we have a starting design and approach. Design: We are going to make Sub-Groups work like user-name does today. Because user-name doesn't pull from the job config because of kerberos and security issues. Approach: 1. I am going to assign the JIRI to my self. 2. I am going to implement a truck (MR2-YERN) solution first. Then back port that solution to branch-1 (MR1) 3. I will try to get the first patch done by end of week next week. Thx
        Hide
        Andrew Noel added a comment -

        Ted provided the java code to my organization out of a request to have a method of better managing the user community workload. This has been huge in governing the workload and taking the pool decision out of the inexperienced user hands and into the hands of the administrators. It allows for a much more granular control.

        Show
        Andrew Noel added a comment - Ted provided the java code to my organization out of a request to have a method of better managing the user community workload. This has been huge in governing the workload and taking the pool decision out of the inexperienced user hands and into the hands of the administrators. It allows for a much more granular control.
        Hide
        Ted Malaska added a comment -

        Here is a sample of extending the existing FairScheduler.

        Please let me know what you think.

        Show
        Ted Malaska added a comment - Here is a sample of extending the existing FairScheduler. Please let me know what you think.

          People

          • Assignee:
            Unassigned
            Reporter:
            Ted Malaska
          • Votes:
            2 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:

              Development