HBase
  1. HBase
  2. HBASE-4721

Retain Delete Markers after Major Compaction

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      There is a need to provide long TTLs for delete markers. This is useful when replicating hbase logs from one cluster to another. The receiving cluster shouldn't compact away the delete markers because the affected key-values might still be on the way.

        Issue Links

          Activity

          Hide
          Prakash Khemani added a comment -

          I think all we need to do is to retain delete markers even after a major compaction.

          Retaining delete markers beyond the ttl of regular KVs doesn't make sense to me. Say the regular KV TTL is 1 day. If we keep a delete marker alive for 2 days then the only KVs that it is going to affect are the ones that are older than 2 days and are already expired.

          If we just retain delete markers after every major compaction then the processing becomes exactly the same as in a minor compaction. Hopefully, will reduce some code complexity.

          Show
          Prakash Khemani added a comment - I think all we need to do is to retain delete markers even after a major compaction. Retaining delete markers beyond the ttl of regular KVs doesn't make sense to me. Say the regular KV TTL is 1 day. If we keep a delete marker alive for 2 days then the only KVs that it is going to affect are the ones that are older than 2 days and are already expired. If we just retain delete markers after every major compaction then the processing becomes exactly the same as in a minor compaction. Hopefully, will reduce some code complexity.
          Hide
          Lars Hofhansl added a comment - - edited

          Is this not covered with HBASE-4536? I.e. just keeping the delete markers will not get you very far, you also have to keep the "deleted" rows (KVs affected by the delete markers).

          Show
          Lars Hofhansl added a comment - - edited Is this not covered with HBASE-4536 ? I.e. just keeping the delete markers will not get you very far, you also have to keep the "deleted" rows (KVs affected by the delete markers).
          Hide
          Prakash Khemani added a comment -

          I had started at a point where I thought I will independently assign ttls to delete markers. But now I have realized that it doesn't make any sense to give a different ttl to the delete-markers. (giving the delete-markers a smaller ttl than the puts will be incorrect. giving them a larger ttl than the puts will be pointless because then the delete-markers will be deleting already expired puts)

          HBASE-4536 will work but only if keep-deleted-kvs flag is set on the column family (or is it table?). Do you think it makes sense to make it the default behavior that regardless of whether point-in-time queries are being supported or not, major compaction will not remove the delete-markers? A delete-marker will only be removed when it expires or when enough put versions accumulate before it.

          Concerns that people have raised if we stopped removing all delete markers in a major compaction
          (1) Space wastage. I am not sure if this is a big concern.
          (2) The bigger issue is that the user will never be able to insert a Put beyond the delete marker. Today, if the user makes a mistake then the admin can go in, delete the puts, do a major compaction, and then the user can reinsert the correct Puts. This workflow will be nullified if we keep delete-markers even after major compaction.
          (3) Today the user doesn't even know that there are delete markers. But that will have to change if we start keeping delete-markers beyond major compactions.

          ===
          I don't get the reasoning behind why we need to keep deleted puts when syncing logs from one cluster to another. The problem that I am concerned about is the following

          (1) Delete marker arrives from the source cluster
          (2) major compaction happens on the target cluster which gets rid of the delete marker
          (3) The deleted put arrives from the source cluster. Now that the delete marker is not there, this put will become visible on the target cluster.

          Show
          Prakash Khemani added a comment - I had started at a point where I thought I will independently assign ttls to delete markers. But now I have realized that it doesn't make any sense to give a different ttl to the delete-markers. (giving the delete-markers a smaller ttl than the puts will be incorrect. giving them a larger ttl than the puts will be pointless because then the delete-markers will be deleting already expired puts) HBASE-4536 will work but only if keep-deleted-kvs flag is set on the column family (or is it table?). Do you think it makes sense to make it the default behavior that regardless of whether point-in-time queries are being supported or not, major compaction will not remove the delete-markers? A delete-marker will only be removed when it expires or when enough put versions accumulate before it. Concerns that people have raised if we stopped removing all delete markers in a major compaction (1) Space wastage. I am not sure if this is a big concern. (2) The bigger issue is that the user will never be able to insert a Put beyond the delete marker. Today, if the user makes a mistake then the admin can go in, delete the puts, do a major compaction, and then the user can reinsert the correct Puts. This workflow will be nullified if we keep delete-markers even after major compaction. (3) Today the user doesn't even know that there are delete markers. But that will have to change if we start keeping delete-markers beyond major compactions. === I don't get the reasoning behind why we need to keep deleted puts when syncing logs from one cluster to another. The problem that I am concerned about is the following (1) Delete marker arrives from the source cluster (2) major compaction happens on the target cluster which gets rid of the delete marker (3) The deleted put arrives from the source cluster. Now that the delete marker is not there, this put will become visible on the target cluster.
          Hide
          Lars Hofhansl added a comment -

          Log based replication won't have any issues with deleted puts, correct.
          I'd be worried about tour concerns (2) and (3) above. In HBASE-4536 I spent a great deal of effort to be able to remove delete markers as soon as possible.

          Interestingly HBASE-4536 would not help with the problem you outline. Column and Version delete markers are version counted (they inherit the version of the last puts that sorts before them), and eventually removed just like normal puts.
          Family delete markers, however, are not sorted in line with puts and are instead removed during a major compaction if (and only if) there are no puts that they affect in the file, which leads to the same problem if puts arrive later during replication, or if the user makes a mistake and places delete markers in the future.

          Do we need to invent a way to undo delete markers?

          Show
          Lars Hofhansl added a comment - Log based replication won't have any issues with deleted puts, correct. I'd be worried about tour concerns (2) and (3) above. In HBASE-4536 I spent a great deal of effort to be able to remove delete markers as soon as possible. Interestingly HBASE-4536 would not help with the problem you outline. Column and Version delete markers are version counted (they inherit the version of the last puts that sorts before them), and eventually removed just like normal puts. Family delete markers, however, are not sorted in line with puts and are instead removed during a major compaction if (and only if) there are no puts that they affect in the file, which leads to the same problem if puts arrive later during replication, or if the user makes a mistake and places delete markers in the future. Do we need to invent a way to undo delete markers?
          Hide
          Todd Lipcon added a comment -

          Log based replication won't have any issues with deleted puts, correct.

          With the log-based replication as it's currently implemented, we don't have any guarantees that actions arrive in the same order on the target cluster as they were initiated on the source cluster, do we? We might have the following sequence:

          • region is hosted on server A on source cluster
          • A row is put (goes in A's HLog)
          • region closes on A, moves to server B
          • A now enters a long GC pause, or goes down (before the log shipping got the newest records)
          • the row is deleted (goes in B's HLog)
          • replication receives the delete from B's HLog before the put from A's HLog

          To prevent this, the log-based replication needs to do something to achieve better transactional ordering when re-applying the edits on the target cluster. Or, we need something like we're discussing which allows the delete to be retained for some period of time.

          Show
          Todd Lipcon added a comment - Log based replication won't have any issues with deleted puts, correct. With the log-based replication as it's currently implemented, we don't have any guarantees that actions arrive in the same order on the target cluster as they were initiated on the source cluster, do we? We might have the following sequence: region is hosted on server A on source cluster A row is put (goes in A's HLog) region closes on A, moves to server B A now enters a long GC pause, or goes down (before the log shipping got the newest records) the row is deleted (goes in B's HLog) replication receives the delete from B's HLog before the put from A's HLog To prevent this, the log-based replication needs to do something to achieve better transactional ordering when re-applying the edits on the target cluster. Or, we need something like we're discussing which allows the delete to be retained for some period of time.
          Hide
          Lars Hofhansl added a comment -

          I think the point Prakash was making is that we only have keep the deletes in the store files (the delete markers), but not the deleted KVs (the puts that would be affected by the deletes), since the replication would be using the logs anyway.

          In your scenario above... As long as we keep the deletes at the target for a while, it won't be a problem if the puts arrive later (both will retain the time stamp from the source).
          That is what Prakash wants to achieve with this.

          Show
          Lars Hofhansl added a comment - I think the point Prakash was making is that we only have keep the deletes in the store files (the delete markers), but not the deleted KVs (the puts that would be affected by the deletes), since the replication would be using the logs anyway. In your scenario above... As long as we keep the deletes at the target for a while, it won't be a problem if the puts arrive later (both will retain the time stamp from the source). That is what Prakash wants to achieve with this.
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK #2500 (See https://builds.apache.org/job/HBase-TRUNK/2500/)
          HBASE-4721 Retain Delete Markers after Major Compaction

          Summary: major compactions try to retain delete markers for the configured
          time interval

          Test Plan:
          added a new test testDeleteMarkerLongevity() in TestStoreScanner

          following tests passed
          TestMemStore
          TestStoreScanner
          TestQueryMatcher
          TestCompaction

          Reviewers: jgray, dhruba, lhofhansl, Karthik, nspiegelberg

          Reviewed By: nspiegelberg

          CC: HBase Diffs Facebook Group, lhofhansl, khemani, nspiegelberg

          Differential Revision: 321

          karthik :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK #2500 (See https://builds.apache.org/job/HBase-TRUNK/2500/ ) HBASE-4721 Retain Delete Markers after Major Compaction Summary: major compactions try to retain delete markers for the configured time interval Test Plan: added a new test testDeleteMarkerLongevity() in TestStoreScanner following tests passed TestMemStore TestStoreScanner TestQueryMatcher TestCompaction Reviewers: jgray, dhruba, lhofhansl, Karthik, nspiegelberg Reviewed By: nspiegelberg CC: HBase Diffs Facebook Group, lhofhansl, khemani, nspiegelberg Differential Revision: 321 karthik : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
          Hide
          Hudson added a comment -

          Integrated in HBase-TRUNK-security #15 (See https://builds.apache.org/job/HBase-TRUNK-security/15/)
          HBASE-4721 Retain Delete Markers after Major Compaction

          Summary: major compactions try to retain delete markers for the configured
          time interval

          Test Plan:
          added a new test testDeleteMarkerLongevity() in TestStoreScanner

          following tests passed
          TestMemStore
          TestStoreScanner
          TestQueryMatcher
          TestCompaction

          Reviewers: jgray, dhruba, lhofhansl, Karthik, nspiegelberg

          Reviewed By: nspiegelberg

          CC: HBase Diffs Facebook Group, lhofhansl, khemani, nspiegelberg

          Differential Revision: 321

          karthik :
          Files :

          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
          • /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
          • /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
          Show
          Hudson added a comment - Integrated in HBase-TRUNK-security #15 (See https://builds.apache.org/job/HBase-TRUNK-security/15/ ) HBASE-4721 Retain Delete Markers after Major Compaction Summary: major compactions try to retain delete markers for the configured time interval Test Plan: added a new test testDeleteMarkerLongevity() in TestStoreScanner following tests passed TestMemStore TestStoreScanner TestQueryMatcher TestCompaction Reviewers: jgray, dhruba, lhofhansl, Karthik, nspiegelberg Reviewed By: nspiegelberg CC: HBase Diffs Facebook Group, lhofhansl, khemani, nspiegelberg Differential Revision: 321 karthik : Files : /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/ScanWildcardColumnTracker.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStore.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java /hbase/trunk/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java

            People

            • Assignee:
              Prakash Khemani
              Reporter:
              Prakash Khemani
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development