[MAPREDUCE-7073] Optimize TokenCache#obtainTokensForNamenodesInternal

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.10.0, 2.8.4, 3.2.0, 3.0.3, 3.1.1, 2.9.2
• Hadoop Flags: Reviewed
• Assignee: Bibin Chundatt
• Reporter: Bibin Chundatt
• Created: 09/Apr/18 09:37
• Updated: 13/Nov/18 11:51
• Resolved: 02/May/18 08:02

Description

FileInputFormat#listStatus is too slow file system cache is disabled.

TokenCache#obtainTokensForNamenodesInternal for every filesystem instance Master.getMasterPrincipal(conf) is caled which reloads YarnConfiguration .
For fileInput with 1k file will reload YarnConfiguration 1k times.

Master.getMasterPrincipal(conf) can be passed for obtainTokensForNamenodesInternal per filesystem call.