[AURORA-1869] Investigate the status update processing overhead

• Type: Task
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: Scheduler
• Epic Link: Scheduler Performance Improvement
• Assignee: Unassigned
• Reporter: Mehrdad Nurolahzade
• Created: 21/Dec/16 20:03
• Updated: 05/May/17 19:47

Description

There is a peculiar similarity pattern between the number of task status update events received from Mesos and the number of JVM threads started by the system (graphview). It seems like a new thread is started every time a status update event is processed.

TaskStatusHandlerImpl is a single-threaded service, therefore it should not instantiate new threads. Looking at status update reasons/results, the majority of status updates are associated with RECONCILIATION and should result in NOOP. Therefore, they should have no impact on the internal workers. The task state machine should short-circuit and return upon realizing that the task’s reported new state corresponds to scheduler’s view.

TaskStateMachine.updateState()

if (stateMachine.getState() == taskState) {
  return new TransitionResult(NOOP, ImmutableSet.of());
}

Given the volume of status update events received upon reconciliation this overhead needs to be avoided, if possible.