[HBASE-27688] HFile splitting occurs during bulkload, the CREATE_TIME_TS of hfileinfo is 0

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.6.0, 3.0.0-alpha-4, 2.4.17, 2.5.4
• Component/s: HFile
• Hadoop Flags: Reviewed
• Assignee: alan.zhao
• Reporter: alan.zhao
• Created: 06/Mar/23 02:45
• Updated: 16/Mar/23 06:34
• Resolved: 15/Mar/23 00:26

Description

If HFile splitting occurs during bulkload, the CREATE_TIME_TS of hfileinfo =0,When HFile is copied after splitting, CREATE_TIME_TS of the original file is not copied。

##BulkLoadHFilesTool.class 
/**
 * Copy half of an HFile into a new HFile.
 */
private static void copyHFileHalf(Configuration conf, Path inFile, Path outFile,
  Reference reference, ColumnFamilyDescriptor familyDescriptor) throws IOException {
  FileSystem fs = inFile.getFileSystem(conf);
  CacheConfig cacheConf = CacheConfig.DISABLED;
  HalfStoreFileReader halfReader = null;
  StoreFileWriter halfWriter = null;
  try {
    。。。
    HFileContext hFileContext = new HFileContextBuilder().withCompression(compression)
      .withChecksumType(StoreUtils.getChecksumType(conf))
      .withBytesPerCheckSum(StoreUtils.getBytesPerChecksum(conf)).withBlockSize(blocksize)
      .withDataBlockEncoding(familyDescriptor.getDataBlockEncoding()).withIncludesTags(true)
      .build();
// TODO .withCreateTime(EnvironmentEdgeManager.currentTime())      

halfWriter = new StoreFileWriter.Builder(conf, cacheConf, fs).withFilePath(outFile)
      .withBloomType(bloomFilterType).withFileContext(hFileContext).build();
    HFileScanner scanner = halfReader.getScanner(false, false, false);
    scanner.seekTo();
    do {
      halfWriter.append(scanner.getCell());
    } while (scanner.next());

    for (Map.Entry<byte[], byte[]> entry : fileInfo.entrySet()) {
      if (shouldCopyHFileMetaKey(entry.getKey())) {
        halfWriter.appendFileInfo(entry.getKey(), entry.getValue());
      }
    }
  } finally {
    。。。
  }
} 


##get lastMajorCompactionTs metric

  lastMajorCompactionTs = this.region.getOldestHfileTs(true);
...
  long now = EnvironmentEdgeManager.currentTime();
  return now - lastMajorCompactionTs;
...
##
public long getOldestHfileTs(boolean majorCompactionOnly) throws IOException {
  long result = Long.MAX_VALUE;
  for (HStore store : stores.values()) {
    Collection<HStoreFile> storeFiles = store.getStorefiles();
   ...
    for (HStoreFile file : storeFiles) {
      StoreFileReader sfReader = file.getReader();
     ...
      result = Math.min(result, reader.getFileContext().getFileCreateTime());
    }
  }
  return result == Long.MAX_VALUE ? 0 : result;
}

 

Issue Links

• links to: GitHub Pull Request #5082
• links to: GitHub Pull Request #5097