[YARN-5192] Limit the number of resources a container can localize

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Chris Trezzo
• Created: 02/Jun/16 02:39
• Updated: 02/Jun/16 02:39

Description

Users will sometimes submit YARN applications where containers localize a large number of resources. This thrashes the YARN local cache on the node managers and can impact other running applications. It would be nice if there was server-side enforcement of localization limits that an admin could configure. For example:

1. Total size of the resources a container can localize.
2. Total number of files a container can localize.
3. The maximum size of an individual resource (i.e. a large fat jar) that a container can localize.

MAPREDUCE-6690 is being worked on to provide a set of MapReduce client-side limits.

Issue Links

• relates to: MAPREDUCE-6690 Limit the number of resources a single map reduce job can submit for localization