[HBASE-28696] Partition BackupSystemTable queries - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 2.6.0
Fix Version/s: 2.7.0, 3.0.0-beta-2, 2.6.1
Component/s: None
Labels:
- pull-request-available

Description

When successfully taking an incremental backup, one of our final steps is to delete bulk load metadata from the system table for the bulk loads that needed to be captured in the given backup. This means that we will basically truncate the entire bulk loads system table in a single batch of the deletes after successfully taking an incremental backup. This logic occurs in BackupSystemTable#deleteBulkLoadedRows:

/*
 * Removes rows recording bulk loaded hfiles from backup table
 * @param lst list of table names
 * @param rows the rows to be deleted
 */
public void deleteBulkLoadedRows(List<byte[]> rows) throws IOException {
  try (Table table = connection.getTable(bulkLoadTableName)) {
    List<Delete> lstDels = new ArrayList<>();
    for (byte[] row : rows) {
      Delete del = new Delete(row);
      lstDels.add(del);
      LOG.debug("orig deleting the row: " + Bytes.toString(row));
    }
    table.delete(lstDels);
    LOG.debug("deleted " + rows.size() + " original bulkload rows");
  }
}

Depending on your usage, one may run tons of bulk loads between backups, so this design is needlessly fragile. We should partition these deletes so that we never erroneously fail a backup due to this.