[TAJO-1508] ResourceTracker does not update workers' resource capacities after the first join

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Component/s: Resource Manager, Worker
• Assignee: Unassigned
• Reporter: Hyunsik Choi
• Created: 02/Apr/15 05:10
• Updated: 04/Aug/15 01:51
• Resolved: 04/Aug/15 01:51

Description

Background

Resource Tracker (RT) registers a worker with its connection info and resource capacity into RMContext when the worker sends the first heartbeat to RT.

Problem

• All heartbeats that each worker sends to RT always include the resource capacity. It is not necessary and inefficient in terms of network costs.

• After the first heartbeat, RT does not update the resource capacity of the worker even though the resource capacity of the same worker is changed. This case is unusual, but it is possible. For example, some workers can rejoin to the Tajo cluster with modified resource capacities. But, RT will still keep previous resource capacity.

Solution

• We should change resource capacity of heartbeat to optional fields.
• Only the first heartbeat of each worker includes the resource capacity of the worker.
• When a resource capacity field of heartbeat is set, RT updates the resource capacity for the worker.

This solution will reduce the size of heartbeat message and will enable RT to update changed resource capacity of workers.

Issue Links

• is related to: TAJO-1214 (Umbrella) Improve task and node failure handling