Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-13696

Monitor fair-scheduler.xml and automatically update/validate jobs submitted to fair-scheduler

    Details

    • Type: Improvement
    • Status: Reopened
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Ensure that jobs are placed into the correct queue according to fair-scheduler.xml. Jobs should be placed into the correct queue, and users should not be able to submit jobs to queues they do not have access to.

      This patch builds on the existing functionality in FairSchedulerShim to route jobs to user-specific queue based on fair-scheduler.xml configuration (leveraging the Yarn QueuePlacementPolicy class). In addition to configuring job routing at session connect (current behavior), the routing is validated per submission to yarn (when impersonation is off). A FileSystemWatcher class is included to monitor changes in the fair-scheduler.xml file (so updates are automatically reloaded when the file pointed to by yarn.scheduler.fair.allocation.file is changed).

        Attachments

        1. HIVE-13696.14.patch
          25 kB
          Reuben Kuhnert
        2. HIVE-13696.13.patch
          27 kB
          Reuben Kuhnert
        3. HIVE-13696.11.patch
          27 kB
          Reuben Kuhnert
        4. HIVE-13696.08.patch
          29 kB
          Reuben Kuhnert
        5. HIVE-13696.06.patch
          46 kB
          Reuben Kuhnert
        6. HIVE-13696.02.patch
          22 kB
          Reuben Kuhnert
        7. HIVE-13696.01.patch
          18 kB
          Reuben Kuhnert

          Issue Links

            Activity

              People

              • Assignee:
                ychena Yongzhi Chen
                Reporter:
                sircodesalot Reuben Kuhnert
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated: