[AURORA-1837] Improve implicit task history pruning

• Type: Task
• Status: Reviewable
• Priority: Minor
• Resolution: Unresolved
• Labels: scheduler
• Epic Link: Scheduler Performance Improvement
• Assignee: Mehrdad Nurolahzade
• Reporter: Reza Motamedi
• Created: 28/Nov/16 22:46
• Updated: 04/Aug/17 12:47

Description

Current implementation of TaskHistoryPrunner registers all inactive tasks upon terminal state change for pruning. TaskHistoryPrunner::registerInactiveTask() uses a delay executor to schedule the process of pruning _task_s. However, we have noticed most of pruning takes place after scheduler recovers from a fail-over.

Modify TaskHistoryPruner to a design similar to JobUpdateHistoryPruner:

1. Instead of registering delay executor's upon terminal task state transitions, have it wake up on preconfigured intervals, find all terminal state tasks that meet pruning criteria and delete them.
2. Make the initial task history pruning delay configurable so that it does not hamper scheduler upon start.

Issue Links

• relates to: AURORA-1804 interaction of job history pruning configuration parameters should be documented