[HBASE-27365] Minimise block addition failures due to no space in bucket cache writers queue by introducing wait time

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha-3
• Fix Version/s: 2.6.0, 2.5.1, 3.0.0-alpha-4, 2.4.15
• Component/s: BucketCache
• Assignee: Rajeshbabu Chintaguntla
• Reporter: Rajeshbabu Chintaguntla
• Created: 12/Sep/22 17:39
• Updated: 24/Oct/22 20:53
• Resolved: 04/Oct/22 23:06

Description

Currently in bucket cache asynchronous caching mechanism introduced where initially the blocks to be cached will be added to queue and writer threads consume the blocks from the queue and write to bucket cache. In case if block writing to bucket cache is slow then there is a chance that queue of writer threads become full and following block additions will be failed. In case of slower storages like s3 might introduce latencies even if we enable bigger sizes of bucket cache using ephemeral storages. So we can allow configurable wait time while adding blocks to queue so that chances of queue free up is possible during the wait time and block addition failures can be minimised. To avoid the performance impact of wait time in regular read paths we can use the wait time mainly during background operations like compactions, flushes or prefetches etc.

Issue Links

• links to: GitHub Pull Request #4805