[HBASE-28878] Introduce Cache for SFT instances created via StoreFileTrackerFactory

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Prathyusha
• Reporter: Prathyusha
• Created: 23/Sep/24 13:15
• Updated: 23/Sep/24 13:19

Description

As part of HBASE-28564 the creation of HStoreFile is made SFT aware and anytime a store file is created, it need SFT instance.
Now with this all the interaction of HStorefiles need SFT instance.
In case of FileBasedStoreFileTracker, each instance of it loads the backed .filelist file and this can be a costly operation in S3

This Jira targets to introduce a cache layer at StoreFileTrackerFactory for SFT instances 
per each
TableName + Region + CF + Mode (Write/ReadOnly mode of SFT)

More detailed thought process around the same here

Every time we create a StoreFileTracker object it will have no state, and so it will either need to go to the filesystem and list the directory or read the tracker file depending on which type it is in order to initialize as soon as we try to use it.
It's fine... because the original code causes IO to happen also.. however,
What do you think about the possibility of reuse? This is a more general question than a comment about this particular call site. Should the StoreFileTrackerFactory cache instances and return the cached instances that match the arguments to StoreFileTrackerFactory.create() rather than make a new instance? Can StoreFileTracker instances be made thread safe so they can be cached and shared?
If we have reuse, and all the relevant filesystem ops go through the StoreFileTracker, then we could potentially save a lot of filesystem or object store IO, because a reused StoreFileTracker would have the ground truth already and would not need to go to the filesystem or object store and do IO in order to e.g. return the StoreFileInfo of a given path.

---

 

Issue Links

• relates to: HBASE-27842 FileBasedStoreFileTracker v2