Details
-
Sub-task
-
Status: Closed
-
Major
-
Resolution: Resolved
-
None
-
None
Description
To optimize a cube with N ready segments, N optimize jobs + 1 checkpoint job are needed, which may occupy too much resources and make other jobs waiting too much time to be scheduled. Thus, job priority and its dynamic changing will be introduced. The idea is as follows:
- Job Fetcher fetches job metadata from database every M seconds.
- A Running Job List is maintained globally
- A Job Priority Queue is maintained globally
- Before fetching job metadata,
- first check the job priority queue and schedule jobs meeting condition C
- then if the size of Running Job List exceeds the max concurrent job limit L, this fetch round will be skipped
- During fetch round,
- the priority of each job in the queue will increase.
- each new job fetched will be assigned with a default priority.
- the highest L jobs will be added to Running Job List and be submitted to Job Execute Thread Pool
- Once a job finishes, it will be removed from Running Job List and notify the Job Fetcher to fetch job metadata
The condition C is as follows:
- job priority should be above a threshold, to filter low priority jobs
- job should wait at least once, to give chance for low priority jobs to be scheduled