[YARN-11578] Fix performance issue of permission check in verifyAndCreateRemoteLogDir

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.3.9
• Fix Version/s: 3.4.0, 3.2.5, 3.3.9
• Component/s: log-aggregation
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.9
• Hadoop Flags: Reviewed
• Assignee: Tamas Domok
• Reporter: Tamas Domok
• Created: 27/Sep/23 09:08
• Updated: 28/Jan/24 00:41
• Resolved: 02/Oct/23 13:22

Description

YARN-10901 introduced a check to avoid a warn message in NN logs in certain situations (when /tmp/logs is not owned by the yarn user), but it adds 3 NameNode calls (create, setpermission, delete) during log aggregation collection, for every NM. Meaning, when a YARN job completes, at the YARN log aggregation phase this check is done for every job, from every NodeManager.

In 30 minutes 4.2 % of all the NameNode calls were due to this in a cluster. "write" calls need a Namesystem writeLock as well, so the impact is bigger.

Issue Links

• is caused by: YARN-10901 Permission checking error on an existing directory in LogAggregationFileController#verifyAndCreateRemoteLogDir
• links to: GitHub Pull Request #6120
• links to: GitHub Pull Request #6142
• links to: GitHub Pull Request #6143