[YARN-11057] NodeManager may generate too many empty log dirs when we configure many log dirs

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.7.7, 3.3.1
• Component/s: log-aggregation, nodemanager
• Target Version/s: 3.3.9
• Assignee: Yao Guangdong
• Reporter: Yao Guangdong
• Created: 31/Dec/21 06:56
• Updated: 29/Nov/22 15:58

Description

  NodeManager may generate too many empty log dirs when we configure many log dirs in NonAggregationLogHandler mode.For example: We have 24 disks, 512G memory，hypothesis that average time cost is 1 min for every container  and average container's size is 4g.Then parallel running containers in one server are 512G / 4G = 128. Every container will generate more than 24 directories in current policy.Then total directories in one week is 128 * 24 * (60 * 24 * 7) = 30 965 760 .This is not conside the tmp directories. Which will consume too many inods in server and affect the disk's io utils.This is because so many inodes will consume too many memory cached in linux.When the memory is not insufficience the cached inodes will remove from the memory.Which will increase the incidence of scan disk and the disk io utils will become high.Actually, this directories only one is used for container logs for every container. The others is empty.So we can delete the empty directories when the job is finished.Which will reduce too many inodes.