[YARN-8774] Memory leak when CapacityScheduler allocates from reserved container with non-default label

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.2.0, 2.8.5
• Fix Version/s: 2.10.0, 3.2.0, 2.9.2, 3.0.4, 3.1.2, 2.8.6
• Component/s: capacityscheduler
• Assignee: Tao Yang
• Reporter: Tao Yang
• Created: 14/Sep/18 13:30
• Updated: 14/Oct/19 15:37
• Resolved: 28/Sep/18 16:34

Description

The cause is that the RMContainerImpl instance of reserved container lost its node label expression, when scheduler reserves containers for non-default node-label requests, it will be wrongly added into LeafQueue#ignorePartitionExclusivityRMContainers and never be removed.

To reproduce this memory leak:
(1) create reserved container
RegularContainerAllocator#doAllocation:  create RMContainerImpl instanceA (nodeLabelExpression="")
LeafQueue#allocateResource:  RMContainerImpl instanceA is put into  LeafQueue#ignorePartitionExclusivityRMContainers
(2) allocate from reserved container
RegularContainerAllocator#doAllocation: create RMContainerImpl instanceB (nodeLabelExpression="test-label")
(3) From now on, RMContainerImpl instanceA will be left in memory (be kept in LeafQueue#ignorePartitionExclusivityRMContainers) forever until RM restarted