Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.4.0, 3.3.9
Description
YARN-10901 introduced a check to avoid a warn message in NN logs in certain situations (when /tmp/logs is not owned by the yarn user), but it adds 3 NameNode calls (create, setpermission, delete) during log aggregation collection, for every NM. Meaning, when a YARN job completes, at the YARN log aggregation phase this check is done for every job, from every NodeManager.
In 30 minutes 4.2 % of all the NameNode calls were due to this in a cluster. "write" calls need a Namesystem writeLock as well, so the impact is bigger.
Attachments
Issue Links
- is caused by
-
YARN-10901 Permission checking error on an existing directory in LogAggregationFileController#verifyAndCreateRemoteLogDir
-
- Resolved
-
- links to