Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-13696

Monitor fair-scheduler.xml and automatically update/validate jobs submitted to fair-scheduler

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Reopened
    • Major
    • Resolution: Unresolved
    • None
    • None
    • None
    • None

    Description

      Ensure that jobs are placed into the correct queue according to fair-scheduler.xml. Jobs should be placed into the correct queue, and users should not be able to submit jobs to queues they do not have access to.

      This patch builds on the existing functionality in FairSchedulerShim to route jobs to user-specific queue based on fair-scheduler.xml configuration (leveraging the Yarn QueuePlacementPolicy class). In addition to configuring job routing at session connect (current behavior), the routing is validated per submission to yarn (when impersonation is off). A FileSystemWatcher class is included to monitor changes in the fair-scheduler.xml file (so updates are automatically reloaded when the file pointed to by yarn.scheduler.fair.allocation.file is changed).

      Attachments

        1. HIVE-13696.01.patch
          18 kB
          Reuben Kuhnert
        2. HIVE-13696.02.patch
          22 kB
          Reuben Kuhnert
        3. HIVE-13696.06.patch
          46 kB
          Reuben Kuhnert
        4. HIVE-13696.08.patch
          29 kB
          Reuben Kuhnert
        5. HIVE-13696.11.patch
          27 kB
          Reuben Kuhnert
        6. HIVE-13696.13.patch
          27 kB
          Reuben Kuhnert
        7. HIVE-13696.14.patch
          25 kB
          Reuben Kuhnert

        Issue Links

          Activity

            People

              ychena Yongzhi Chen
              sircodesalot Reuben Kuhnert
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated: