[HADOOP-12876] [Azure Data Lake] Support for process level FileStatus cache to optimize GetFileStatus frequent operations

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: fs, fs/adl, tools
• Target Version/s: 3.4.0
• Assignee: Vishwajeet Dusane
• Reporter: Vishwajeet Dusane
• Created: 03/Mar/16 11:24
• Updated: 28/Apr/21 19:31
• Resolved: 28/Apr/21 19:31

Description

Add support to cache GetFileStatus and ListStatus response locally for limited period of time. Local cache for limited period of time would optimize number of calls for GetFileStatus operation.
One of the example where local limited period cache would be useful - terasort ListStatus on input directory follows with GetFileStatus operation on each file within directory. For 2048 input files in a directory would save 2048 GetFileStatus calls during start up (Using the ListStatus response to cache FileStatus instances).

Issue Links

• depends upon: HADOOP-15038 Abstract MetadataStore in S3Guard into a common module.
• is related to: HADOOP-13452 S3Guard: Implement access policy for intra-client consistency with in-memory metadata store.
• is related to: HADOOP-13321 Deprecate FileSystem APIs that promote inefficient call patterns.