[HUDI-1] Design and Implement embedded timeline service to cache filesystem view to reduce listStatus calls

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: performance, writer-core
• Labels: pull-request-available
• Assignee: Balaji Varadarajan
• Reporter: Balaji Varadarajan
• Created: 04/Mar/19 20:26
• Updated: 03/Nov/23 22:28
• Resolved: 29/May/19 23:36

Description

Currently, Hudi writers repeatedly list partitions to create file-system views in executors. This task addresses the reductions in listStatus name-node calls in Hudi 2.0 writers by taking advantage of MVCC view of HUDI and caching file-system view and reusing them.

An embedded file-system view server on driver will be preloaded with the view. It will act as a cache and service File-system view calls from executors.

 

https://github.com/uber/hudi/issues/433

https://github.com/uber/hudi/issues/269

 

 

Issue Links

• links to: GitHub Pull Request #9983