[AURORA-117] Scheduler performance issues with very large jobs

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: Reliability, Scheduler
• Labels: O1
• Assignee: Bill Farner
• Reporter: Bill Farner
• Created: 25/Jan/14 04:13
• Updated: 27/Mar/14 17:20
• Resolved: 19/Feb/14 21:27

Description

The scheduler tends to have performance issues when scheduling very large jobs. We've observed this with jobs exceeding 2000 instances. The TaskScheduler thread tends to consume a large amount of CPU (100%, limited by the global storage lock). Current hypothesis is that the majority of the time is spent satisfying diversity constraints (rack, machine), which require expensive queries.

Issue Links

• is related to: AURORA-139 Scheduling performance could be vastly improved by caching job information