[SPARK-41254] YarnAllocator.rpIdToYarnResource map is not properly updated

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.1.0, 3.2.0, 3.3.1
• Fix Version/s: 3.2.4, 3.3.2, 3.4.0
• Component/s: YARN
• Assignee: Zhang Liang
• Reporter: Zhang Liang
• Created: 24/Nov/22 13:11
• Updated: 28/Nov/22 14:26
• Resolved: 28/Nov/22 14:26

Description

Log messages "INFO YarnAllocator: Resource profile 0 doesn't exist, adding it" repeats multiple times in yarn stderr.

The log should be outputted only once because it happens in YarnAllocator.createYarnResourceForResourceProfile on a default ResourceProfile

After digging into the code, I found a bug caused by misleading usage of ConcurrentHashMap.contains

Issue Links

• links to: [Github] Pull Request #38790 (CavemanIV)