We can keep the existing column level definition and enforcement mechanism and extend it to look for a TTL cell tag during compaction. If one is found, it can override the CF setting. TTL overrides can be passed up to the server in an operation attribute.

This is the same strategy used for ACLs on HBASE-7662

I like the idea!

Great idea.

Moving to 0.98.2

This is something I'd like to see so assigning to myself to do in the 0.98.5 timeframe.

Lars Hofhansl mentioned this on user@:

Subject: Re: Coprocessor beacuse of TTL expired?
[...]
HBASE-11054 is about delete trackers (i.e. when/how delete markers are enforced), TTL is handled differently. One could ask why these are handled differently of course.

We could try moving TTL logic into the delete tracker. Would need to check how the stacking works after HBASE-10885.

Move out to .6. Please see the thread 'Thinking about rolling 0.98.5RC0 on August 4' on dev@ for context.

We could try moving TTL logic into the delete tracker. Would need to check how the stacking works after HBASE-10885.

I think that would be cool. In fact, why not express all the ScanQueryMatcher logic with filters (that's something that our Jesse Yates suggested a long time ago). That would (a) allow users to easily customize the logic and (b) get rid of a whole bunch of custom framework logic.
Imagine a FilterList and deals with deletes, TTL, MinVersions, etc, each implemented as a clean, separate filter. How cool would that be?

In fact, why not express all the ScanQueryMatcher logic with filters (that's something that our Jesse Yates suggested a long time ago).

That would be cool, but perhaps a follow up issue? I.e. move TTL tracking to the SQM as a subtask here, add per cell logic as another, commit both, then complete the suggested larger scope refactor.

each implemented as a clean, separate filter

HBASE-11084 is about that.

HBASE-11084 is about that

Nice... Missed that.

That would be cool, but perhaps a follow up issue? I.e. move TTL tracking to the SQM as a subtask here, add per cell logic as another, commit both, then complete the suggested larger scope refactor.

Let's do that. And use HBASE-11084 for the larger refactor.

Looks like HBASE-11546 will make .6, let's move this to .7 out of an abundance of caution.

Under active review. Moving out of 0.98.7, should land in 0.98.8

Thanks for the reviews and the 1s. I will commit this to 0.98 just after the 0.98.8 release. Moving to 0.98.9.

All subtasks resolved

The work ended up bundled into a combined patch on HBASE-11764, see that issue for patches, but the changes were committed using the id of this issue. A bit weird, sorry about that, subtasks weren't really needed.

FAILURE: Integrated in HBase-TRUNK #5872 (See https://builds.apache.org/job/HBase-TRUNK/5872/)
HBASE-10560 Per cell TTLs (apurtell: rev 09cd3d7bfb9f4c4b279a74441d4059da6b8177d2)

• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
• hbase-shell/src/test/ruby/hbase/table_test.rb
• hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java
• hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithTTLs.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
• hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java
• hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTags.java
• hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
• hbase-common/src/main/java/org/apache/hadoop/hbase/TagType.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CellCreator.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java
• hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java
• hbase-client/src/main/java/org/apache/hadoop/hbase/client/Increment.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/GetClosestRowBeforeTracker.java
• hbase-shell/src/main/ruby/hbase/table.rb
• hbase-client/src/main/java/org/apache/hadoop/hbase/client/Append.java
• hbase-client/src/main/java/org/apache/hadoop/hbase/client/Put.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
• hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TsvImporterMapper.java

SUCCESS: Integrated in HBase-1.0 #546 (See https://builds.apache.org/job/HBase-1.0/546/)
HBASE-10560 Per cell TTLs (apurtell: rev 004e977ba0009fc8978d842eb65933a62fdf1173)

• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
• hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
• hbase-client/src/main/java/org/apache/hadoop/hbase/client/Increment.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
• hbase-common/src/main/java/org/apache/hadoop/hbase/TagType.java
• hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithTTLs.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CellCreator.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java
• hbase-shell/src/test/ruby/hbase/table_test.rb
• hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
• hbase-client/src/main/java/org/apache/hadoop/hbase/client/Append.java
• hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTags.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TsvImporterMapper.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
• hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java
• hbase-client/src/main/java/org/apache/hadoop/hbase/client/Put.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/GetClosestRowBeforeTracker.java
• hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
• hbase-shell/src/main/ruby/hbase/table.rb

SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #689 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/689/)
HBASE-10560 Per cell TTLs (apurtell: rev 869c5665cd4cc0cd7994933e839154002f682f44)

• hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java
• hbase-common/src/main/java/org/apache/hadoop/hbase/TagType.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TsvImporterMapper.java
• hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTags.java
• hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
• hbase-shell/src/test/ruby/hbase/table_test.rb
• hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/GetClosestRowBeforeTracker.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
• hbase-shell/src/main/ruby/hbase/table.rb
• hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CellCreator.java
• hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java
• hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
• hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
• hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithTTLs.java

SUCCESS: Integrated in HBase-0.98 #722 (See https://builds.apache.org/job/HBase-0.98/722/)
HBASE-10560 Per cell TTLs (apurtell: rev 869c5665cd4cc0cd7994933e839154002f682f44)

• hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithTTLs.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CellCreator.java
• hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
• hbase-common/src/main/java/org/apache/hadoop/hbase/TagType.java
• hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/GetClosestRowBeforeTracker.java
• hbase-shell/src/main/ruby/hbase/table.rb
• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
• hbase-shell/src/test/ruby/hbase/table_test.rb
• hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TsvImporterMapper.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
• hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTags.java
• hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java
• hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
• hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java

Closing this issue after 1.0.0 release.

That is from Phoenix thread

By the way a cell TTL is not a future timestamp, it's the desired lifetime of the cell in milliseconds. Cell TTLs can't cause a cell to live beyond any TTL defined in the table schema (due to TTL handling optimizations at the Store level), so it would make sense to throw an exception if we detect a user is trying to do that.

I do not think HBase checks if the cell TTL is greater than CF TTL.

Yes, HBase ignores cell TTL higher than CF TTL, but does not throw any exception. May be it is OK.

That was a suggestion I made for Phoenix.

We have it documented in the section of the HBase book that talks about cell TTLs that cell TTLs cannot extend the life of a cell if there is a CF-level TTL setting in effect.