[TAJO-743] Change the default resource allocation policy of leaf tasks

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.8.0, 0.9.0
• Fix Version/s: 0.8.0, 0.9.0
• Component/s: Resource Manager
• Assignee: Jinho Kim
• Reporter: Jinho Kim
• Created: 09/Apr/14 07:08
• Updated: 10/Apr/14 06:30
• Resolved: 10/Apr/14 06:06

Description

Currently, resource allocation is calculated by memory base. If a machine have a large memory, in default settings, heavy disk IO per disk is usually caused by high task concurrency. However, it is likely to seem to be problematic.

When i tested the leaf task scan by 2(concurrency of SATA disk), the performance was better. if you have SAS Storage or SSD, you can increase the disk concurrency. This patch changes the default resource allocation policy to use disk resource.

The following configs have been available so far:

• tajo.worker.resource.disks - available disk resource of each worker
• tajo.task.disk-slot.default - how many disk resource is consumed per task

Below config is newly introduced in this patch

• tajo.worker.resource.dfs-dir-aware - it can be true/false. If it is true, each worker uses the number of HDFS datanode's data dirs in the worker as the disk resource. So, tajo.worker.resource.disks is ignored.