Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
2.6.0, 3.0.0-beta-1, 4.0.0-alpha-1, 2.7.0
-
None
Description
Noticed that TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning was flakey, failing whenever the block eviction happened while prefetch was still ongoing.
In the test, we pass an instance of BucketCache directly to the cache config, so the test is actually placing both data and meta blocks in the bucket cache. So sometimes, the test call BucketCache.notifyFileCachingCompleted after the it has already evicted two blocks.
Inside BucketCache.notifyFileCachingCompleted, we iterate through the backingMap entry set, counting number of blocks for the given file. Then, to consider whether the file is fully cached or not, we do the following validation:
if (dataBlockCount == count.getValue() || totalBlockCount == count.getValue()) {
LOG.debug("File {} has now been fully cached.", fileName);
fileCacheCompleted(fileName, size);
}
But the test generates 57 total blocks, 55 data and 2 meta blocks. It evicts two blocks and asserts that the file hasn't been considered fully cached. When these evictions happen while prefetch is still going, we'll pass that check, as the the number of blocks for the file in the backingMap would still be 55, which is what we pass as dataBlockCount.
As BucketCache is intended for storing data blocks only, I believe we should make sure BucketCache.notifyFileCachingCompleted only accounts for data blocks. Also, the TestBucketCachePersister.testPrefetchBlockEvictionWhilePrefetchRunning should be updated to consistently reproduce the eviction concurrent to the prefetch.
Attachments
Issue Links
- is related to
-
-
- In Progress
-
- links to