[YARN-4540] Yarn property "yarn.nodemanager.localizer.cache.cleanup.interval-ms" not working as expected

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 2.7.1
• Component/s: nodemanager, yarn

Environment

Ambari – Version 2.1.2.1
YARN - Version 2.7.1.2.3
HDP - Version 2.3.2.0

• Assignee: Unassigned
• Reporter: Mark S
• Created: 04/Jan/16 17:55
• Updated: 12/Jan/16 02:16

Description

I manually add specific YARN configuration to reduce my YARN cache size, however it does not get updated for the interval specified by:

yarn.nodemanager.localizer.cache.cleanup.interval-ms

I did notice that eventually my server did reduce my YARN cache size over the weekend.

Set Values (Should update size in 1 minute):

yarn.nodemanager.localizer.cache.target-size-mb=4096
yarn.nodemanager.localizer.cache.cleanup.interval-ms=60000

Default values (Should update size in 10 minutes):

yarn.nodemanager.localizer.cache.target-size-mb=10240
yarn.nodemanager.localizer.cache.cleanup.interval-ms=600000

Confirming YARN cache size

date && du -m /hadoop/yarn | sort -nr | head -n 20
#date && du -m . | sort -nr | head -n 20

See also:

• https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-common/yarn-default.xml
• http://hortonworks.com/blog/resource-localization-in-yarn-deep-dive/