[HUDI-104] Replace ThreadLocal with Resource pool when caching resource type classes

• Type: Improvement
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Component/s: Common Core, performance, timeline-server
• Assignee: Unassigned
• Reporter: Balaji Varadarajan
• Created: 19/Apr/19 04:17
• Updated: 16/Jan/22 06:18

Description

This is another comment that stemmed from Timeline Server code-review.

 

Notes from PR comments:

'''RandomAccessFile is not threadsafe. Hence, we would have to create separate RandomAccessFileObjects for concurrent access. Agree, the current approach of opening a new file per get() call is slow. I have made changes to use ThreadLocal objects. This should reduce the open() calls greatly but for a long running timeline service, there could be open handles getting accumulated per thread. A better approach would be to use Resource Pool but needs to be config tuned and tested. There is another place (ThreadLocal where we might require similar treatment. Will open a ticket to address it. If current perf testing points to the direction of Resource pool, will address it as part of this PR. Otherwise, we can take this in subsequent PR'''