I had started at a point where I thought I will independently assign ttls to delete markers. But now I have realized that it doesn't make any sense to give a different ttl to the delete-markers. (giving the delete-markers a smaller ttl than the puts will be incorrect. giving them a larger ttl than the puts will be pointless because then the delete-markers will be deleting already expired puts)
HBASE-4536 will work but only if keep-deleted-kvs flag is set on the column family (or is it table?). Do you think it makes sense to make it the default behavior that regardless of whether point-in-time queries are being supported or not, major compaction will not remove the delete-markers? A delete-marker will only be removed when it expires or when enough put versions accumulate before it.
Concerns that people have raised if we stopped removing all delete markers in a major compaction
(1) Space wastage. I am not sure if this is a big concern.
(2) The bigger issue is that the user will never be able to insert a Put beyond the delete marker. Today, if the user makes a mistake then the admin can go in, delete the puts, do a major compaction, and then the user can reinsert the correct Puts. This workflow will be nullified if we keep delete-markers even after major compaction.
(3) Today the user doesn't even know that there are delete markers. But that will have to change if we start keeping delete-markers beyond major compactions.
I don't get the reasoning behind why we need to keep deleted puts when syncing logs from one cluster to another. The problem that I am concerned about is the following
(1) Delete marker arrives from the source cluster
(2) major compaction happens on the target cluster which gets rid of the delete marker
(3) The deleted put arrives from the source cluster. Now that the delete marker is not there, this put will become visible on the target cluster.