[HDFS-16967] RBF: File based state stores should allow concurrent access to the records

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: rbf
• Labels: pull-request-available
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Viraj Jasani
• Reporter: Viraj Jasani
• Created: 30/Mar/23 07:45
• Updated: 28/Jan/24 07:10
• Resolved: 04/Apr/23 13:41

Description

File based state store implementations (StateStoreFileImpl and StateStoreFileSystemImpl) should allow updating as well as reading of the state store records concurrently rather than serially. Concurrent access to the record files on the hdfs based store seems to be improving the state store cache loading performance by more than 10x.

For instance, in order to maintain data integrity, when any mount table record(s) is updated, the cache is reloaded. This reload operation seems to be able to gain significant performance improvement by the concurrent access of the mount table records.

Issue Links

• links to: GitHub Pull Request #5523