[HBASE-23149] hbase shouldPerformMajorCompaction logic is not correct

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.4.9
• Fix Version/s: 1.6.0, 1.4.12, 1.3.7
• Component/s: Compaction
• Assignee: Jacky Lau
• Reporter: Jacky Lau
• Created: 11/Oct/19 10:01
• Updated: 11/Nov/19 17:20
• Resolved: 24/Oct/19 09:17

Description

we can know that the major compaction is skipping  from the below regionserver's log, but it is compacting that region.

and read the code and find it is not correct  and i add mark  "/*** ***/" below

 

public boolean shouldPerformMajorCompaction(final Collection<StoreFile> filesToCompact)
throws IOException {
if (lowTimestamp > 0L && lowTimestamp < (now - mcTime)) {
if (filesToCompact.size() == 1) {
if (sf.isMajorCompaction() && (cfTTL == Long.MAX_VALUE || oldest < cfTTL)) {
float blockLocalityIndex =
sf.getHDFSBlockDistribution().getBlockLocalityIndex(
RSRpcServices.getHostname(comConf.conf, false));
if (blockLocalityIndex < comConf.getMinLocalityToForceCompact())

{ result = true; }

else

{ LOG.debug("Skipping major compaction of " + regionInfo + " because one (major) compacted file only, oldestTime " + oldest + "ms is < TTL=" + cfTTL + " and blockLocalityIndex is " + blockLocalityIndex + " (min " + comConf.getMinLocalityToForceCompact() + ")"); }

} else if (cfTTL != HConstants.FOREVER && oldest > cfTTL)

{ LOG.debug("Major compaction triggered on store " + regionInfo + ", because keyvalues outdated; time since last major compaction " + (now - lowTimestamp) + "ms"); result = true; }

} else

{ LOG.debug("Major compaction triggered on store " + regionInfo + "; time since last major compaction " + (now - lowTimestamp) + "ms"); }

result = true;  /****here it is not right, it should be move to the above *****/
}
return result;
}

 

2019-09-27 09:09:35,960 DEBUG [st129,16020,1568236573216_ChoreService_1] compactions.RatioBasedCompactionPolicy: Skipping major compaction of 100E_POINT_point_2ddata_z3_geom_GpsTime_v6,\x17,1568215725799.413a563092544e8df480fd601b2de71b. because one (major) compacted file only, oldestTime 3758085589ms is < TTL=9223372036854775807 and blockLocalityIndex is 1.0 (min 0.0)
2019-09-27 09:09:35,961 DEBUG [st129,16020,1568236573216_ChoreService_1] compactions.SortedCompactionPolicy: Selecting compaction from 1 store files, 0 compacting, 1 eligible, 100 blocking
2019-09-27 09:09:35,961 DEBUG [st129,16020,1568236573216_ChoreService_1] regionserver.HStore: 413a563092544e8df480fd601b2de71b - d: Initiating major compaction (all files)
2019-09-27 09:09:35,961 DEBUG [st129,16020,1568236573216_ChoreService_1] regionserver.CompactSplitThread: Large Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@4b5582f1; Because: CompactionChecker requests major compaction; use default priority; compaction_queue=(1:0), split_queue=0, merge_queue=0
2019-09-27 09:09:35,961 INFO [regionserver/st129/10.3.72.129:16020-longCompactions-1568236575579] regionserver.HRegion: Starting compaction on d in region 100E_POINT_point_2ddata_z3_geom_GpsTime_v6,\x17,1568215725799.413a563092544e8df480fd601b2de71b.
2019-09-27 09:09:35,961 INFO [regionserver/st129/10.3.72.129:16020-longCompactions-1568236575579] regionserver.HStore: Starting compaction of 1 file(s) in d of 100E_POINT_point_2ddata_z3_geom_GpsTime_v6,\x17,1568215725799.413a563092544e8df480fd601b2de71b. into tmpdir=hdfs://st129:8020/hbase/data/default/100E_POINT_point_2ddata_z3_geom_GpsTime_v6/413a563092544e8df480fd601b2de71b/.tmp, totalSize=5.1 G
2019-09-27 09:09:35,961 DEBUG [regionserver/st129/10.3.72.129:16020-longCompactions-1568236575579] compactions.Compactor: Compacting hdfs://st129:8020/hbase/data/default/100E_POINT_point_2ddata_z3_geom_GpsTime_v6/413a563092544e8df480fd601b2de71b/d/3b4080f9b6f149e1b0a476058c8564e6, keycount=83914030, bloomtype=NONE, size=5.1 G, encoding=FAST_DIFF, compression=SNAPPY, seqNum=2621061, earliestPutTs=1565788490371

Issue Links

• incorporates: HBASE-23200 incorrect description in SortedCompactionPolicy.getNextMajorCompactTime
• is related to: HBASE-19199 RatioBasedCompactionPolicy#shouldPerformMajorCompaction() always return true when only one file needs to compact
• links to: GitHub Pull Request #744