[HIVE-13696] Monitor fair-scheduler.xml and automatically update/validate jobs submitted to fair-scheduler

• Type: Improvement
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Assignee: Yongzhi Chen
• Reporter: Reuben Kuhnert
• Created: 05/May/16 18:19
• Updated: 27/Feb/24 22:23

Description

Ensure that jobs are placed into the correct queue according to fair-scheduler.xml. Jobs should be placed into the correct queue, and users should not be able to submit jobs to queues they do not have access to.

This patch builds on the existing functionality in FairSchedulerShim to route jobs to user-specific queue based on fair-scheduler.xml configuration (leveraging the Yarn QueuePlacementPolicy class). In addition to configuring job routing at session connect (current behavior), the routing is validated per submission to yarn (when impersonation is off). A FileSystemWatcher class is included to monitor changes in the fair-scheduler.xml file (so updates are automatically reloaded when the file pointed to by yarn.scheduler.fair.allocation.file is changed).

Issue Links

• breaks: HIVE-15615 Fix unit tests failures caused by HIVE-13696
• links to: Review Board