[HBASE-16454] Compactor's shipping logic decision should be based on the current store's block size

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Hadoop Flags: Reviewed
• Assignee: ramkrishna.s.vasudevan
• Reporter: ramkrishna.s.vasudevan
• Created: 19/Aug/16 07:28
• Updated: 23/Aug/16 09:43
• Resolved: 23/Aug/16 05:36

Description

Currently we do call KeyValueScanner.shipped() if the size of the KVs compacted reaches certain threshold

     if (kvs != null && bytesWrittenProgressForShippedCall > shippedCallSizeLimit) {
            kvs.shipped();
            bytesWrittenProgressForShippedCall = 0;
}

Where the shippedCallSizeLimit is

    long shippedCallSizeLimit = (long) minFilesToCompact * HConstants.DEFAULT_BLOCKSIZE;

Instead it is better if it is based on the actual configured block size for that store. It can still have the multiplying factore of 'minFilesToCompact ' but having default store size would mean that if the store has smaller block sizes configured we hold on to more blocks and those will not get evicted.