[MESOS-17] Hadoop executors killed while tasks in COMMIT_PENDING

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.10.0
• Component/s: containerization
• Labels: hadoop lxc

Environment

LXC isolation module, Hadoop framework

• Assignee: Unassigned
• Reporter: Charles Reiss
• Created: 06/Jun/11 21:12
• Updated: 26/Apr/17 16:53
• Resolved: 07/Feb/13 21:54

Description

The Hadoop framework considers tasks finished when they are in the COMMIT_PENDING state. When using the LXC isolation module, this can cause the Hadoop executor's memory allocation to be reduced before the task actually commits. When this happens, the Hadoop executor is sometimes killed for exceeding its memory allocation, leaving the tasks stalled until the master detects the lost task tracker by timeout.