Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
None
-
None
-
None
-
Reviewed
Description
Currently we do call KeyValueScanner.shipped() if the size of the KVs compacted reaches certain threshold
if (kvs != null && bytesWrittenProgressForShippedCall > shippedCallSizeLimit) { kvs.shipped(); bytesWrittenProgressForShippedCall = 0; }
Where the shippedCallSizeLimit is
long shippedCallSizeLimit = (long) minFilesToCompact * HConstants.DEFAULT_BLOCKSIZE;
Instead it is better if it is based on the actual configured block size for that store. It can still have the multiplying factore of 'minFilesToCompact ' but having default store size would mean that if the store has smaller block sizes configured we hold on to more blocks and those will not get evicted.