[MAPREDUCE-2011] Reduce number of getFileStatus call made from every task(TaskDistributedCache) setup

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: distributed-cache
• Assignee: Unassigned
• Reporter: Koji Noguchi
• Created: 14/Aug/10 00:09
• Updated: 11/Jan/16 17:18
• Resolved: 11/Jan/16 17:18

Description

On our cluster, we had jobs with 20 dist cache and very short-lived tasks resulting in 500 map tasks launched per second resulting in 10,000 getFileStatus calls to the namenode. Namenode can handle this but asking to see if we can reduce this somehow.