Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-23149

hbase shouldPerformMajorCompaction logic is not correct

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.9
    • Fix Version/s: 1.6.0, 1.4.12, 1.3.7
    • Component/s: Compaction
    • Labels:
      None

      Description

      we can know that the major compaction is skipping  from the below regionserver's log, but it is compacting that region.

      and read the code and find it is not correct  and i add mark  "/*** ***/" below

       

      public boolean shouldPerformMajorCompaction(final Collection<StoreFile> filesToCompact)
      throws IOException {
      if (lowTimestamp > 0L && lowTimestamp < (now - mcTime)) {
      if (filesToCompact.size() == 1) {
      if (sf.isMajorCompaction() && (cfTTL == Long.MAX_VALUE || oldest < cfTTL)) {
      float blockLocalityIndex =
      sf.getHDFSBlockDistribution().getBlockLocalityIndex(
      RSRpcServices.getHostname(comConf.conf, false));
      if (blockLocalityIndex < comConf.getMinLocalityToForceCompact())

      { result = true; }

      else

      { LOG.debug("Skipping major compaction of " + regionInfo + " because one (major) compacted file only, oldestTime " + oldest + "ms is < TTL=" + cfTTL + " and blockLocalityIndex is " + blockLocalityIndex + " (min " + comConf.getMinLocalityToForceCompact() + ")"); }

      } else if (cfTTL != HConstants.FOREVER && oldest > cfTTL)

      { LOG.debug("Major compaction triggered on store " + regionInfo + ", because keyvalues outdated; time since last major compaction " + (now - lowTimestamp) + "ms"); result = true; }

      } else

      { LOG.debug("Major compaction triggered on store " + regionInfo + "; time since last major compaction " + (now - lowTimestamp) + "ms"); }

      result = true;  /****here it is not right, it should be move to the above *****/
      }
      return result;
      }

       

      2019-09-27 09:09:35,960 DEBUG [st129,16020,1568236573216_ChoreService_1] compactions.RatioBasedCompactionPolicy: Skipping major compaction of 100E_POINT_point_2ddata_z3_geom_GpsTime_v6,\x17,1568215725799.413a563092544e8df480fd601b2de71b. because one (major) compacted file only, oldestTime 3758085589ms is < TTL=9223372036854775807 and blockLocalityIndex is 1.0 (min 0.0)
      2019-09-27 09:09:35,961 DEBUG [st129,16020,1568236573216_ChoreService_1] compactions.SortedCompactionPolicy: Selecting compaction from 1 store files, 0 compacting, 1 eligible, 100 blocking
      2019-09-27 09:09:35,961 DEBUG [st129,16020,1568236573216_ChoreService_1] regionserver.HStore: 413a563092544e8df480fd601b2de71b - d: Initiating major compaction (all files)
      2019-09-27 09:09:35,961 DEBUG [st129,16020,1568236573216_ChoreService_1] regionserver.CompactSplitThread: Large Compaction requested: org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext@4b5582f1; Because: CompactionChecker requests major compaction; use default priority; compaction_queue=(1:0), split_queue=0, merge_queue=0
      2019-09-27 09:09:35,961 INFO [regionserver/st129/10.3.72.129:16020-longCompactions-1568236575579] regionserver.HRegion: Starting compaction on d in region 100E_POINT_point_2ddata_z3_geom_GpsTime_v6,\x17,1568215725799.413a563092544e8df480fd601b2de71b.
      2019-09-27 09:09:35,961 INFO [regionserver/st129/10.3.72.129:16020-longCompactions-1568236575579] regionserver.HStore: Starting compaction of 1 file(s) in d of 100E_POINT_point_2ddata_z3_geom_GpsTime_v6,\x17,1568215725799.413a563092544e8df480fd601b2de71b. into tmpdir=hdfs://st129:8020/hbase/data/default/100E_POINT_point_2ddata_z3_geom_GpsTime_v6/413a563092544e8df480fd601b2de71b/.tmp, totalSize=5.1 G
      2019-09-27 09:09:35,961 DEBUG [regionserver/st129/10.3.72.129:16020-longCompactions-1568236575579] compactions.Compactor: Compacting hdfs://st129:8020/hbase/data/default/100E_POINT_point_2ddata_z3_geom_GpsTime_v6/413a563092544e8df480fd601b2de71b/d/3b4080f9b6f149e1b0a476058c8564e6, keycount=83914030, bloomtype=NONE, size=5.1 G, encoding=FAST_DIFF, compression=SNAPPY, seqNum=2621061, earliestPutTs=1565788490371

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                jackylau jackylau
                Reporter:
                jackylau jackylau
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: