[TEZ-2217] The min-held-containers being released prematurely

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.6.0, 0.5.3, 0.7.0
• Fix Version/s: 0.5.4
• Hadoop Flags: Reviewed
• Assignee: Bikas Saha
• Reporter: Gopal Vijayaraghavan
• Created: 20/Mar/15 22:51
• Updated: 30/Jun/15 04:52
• Resolved: 26/Mar/15 18:04

Description

The min-held containers constraint is respected during query idle times, but is not respected when a query is actually in motion.

The AM releases unused containers during dag execution without checking for min-held containers.

2015-03-20 15:41:53,475 INFO [DelayedContainerManager] rm.YarnTaskSchedulerService: Container's idle timeout expired. Releasing container, containerId=container_1424502260528_1348_01_000013, containerExpiryTime=1426891313264, idleTimeoutMin=5000
2015-03-20 15:41:53,475 INFO [DelayedContainerManager] rm.YarnTaskSchedulerService: Releasing unused container: container_1424502260528_1348_01_000013

This is actually useful only after the AM has received a soft pre-emption message, doing it on an idle cluster slows down one of the most common query patterns in BI systems.

create temporary table smalltable as ...; 
select ... bigtable JOIN smalltable ON ...;

The smaller query in the beginning throws away the pre-warmed capacity.