Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.98.0
    • Fix Version/s: 1.0.0, 2.0.0, 0.98.9
    • Component/s: None
    • Labels:
      None

      Description

      Now that we have cell tags, we can optionally store TTLs per cell.

        Issue Links

          Activity

          Hide
          Andrew Purtell added a comment -

          We can keep the existing column level definition and enforcement mechanism and extend it to look for a TTL cell tag during compaction. If one is found, it can override the CF setting. TTL overrides can be passed up to the server in an operation attribute.

          This is the same strategy used for ACLs on HBASE-7662

          Show
          Andrew Purtell added a comment - We can keep the existing column level definition and enforcement mechanism and extend it to look for a TTL cell tag during compaction. If one is found, it can override the CF setting. TTL overrides can be passed up to the server in an operation attribute. This is the same strategy used for ACLs on HBASE-7662
          Hide
          Jean-Marc Spaggiari added a comment -

          I like the idea!

          Show
          Jean-Marc Spaggiari added a comment - I like the idea!
          Hide
          Lars Hofhansl added a comment -

          Great idea.

          Show
          Lars Hofhansl added a comment - Great idea.
          Hide
          Andrew Purtell added a comment -

          Moving to 0.98.2

          Show
          Andrew Purtell added a comment - Moving to 0.98.2
          Hide
          Andrew Purtell added a comment -

          This is something I'd like to see so assigning to myself to do in the 0.98.5 timeframe.

          Show
          Andrew Purtell added a comment - This is something I'd like to see so assigning to myself to do in the 0.98.5 timeframe.
          Hide
          Andrew Purtell added a comment -

          Lars Hofhansl mentioned this on user@:

          Subject: Re: Coprocessor beacuse of TTL expired?
          [...]
          HBASE-11054 is about delete trackers (i.e. when/how delete markers are enforced), TTL is handled differently. One could ask why these are handled differently of course.

          We could try moving TTL logic into the delete tracker. Would need to check how the stacking works after HBASE-10885.

          Show
          Andrew Purtell added a comment - Lars Hofhansl mentioned this on user@: Subject: Re: Coprocessor beacuse of TTL expired? [...] HBASE-11054 is about delete trackers (i.e. when/how delete markers are enforced), TTL is handled differently. One could ask why these are handled differently of course. We could try moving TTL logic into the delete tracker. Would need to check how the stacking works after HBASE-10885 .
          Hide
          Andrew Purtell added a comment -

          Move out to .6. Please see the thread 'Thinking about rolling 0.98.5RC0 on August 4' on dev@ for context.

          Show
          Andrew Purtell added a comment - Move out to .6. Please see the thread 'Thinking about rolling 0.98.5RC0 on August 4' on dev@ for context.
          Hide
          Lars Hofhansl added a comment -

          We could try moving TTL logic into the delete tracker. Would need to check how the stacking works after HBASE-10885.

          I think that would be cool. In fact, why not express all the ScanQueryMatcher logic with filters (that's something that our Jesse Yates suggested a long time ago). That would (a) allow users to easily customize the logic and (b) get rid of a whole bunch of custom framework logic.
          Imagine a FilterList and deals with deletes, TTL, MinVersions, etc, each implemented as a clean, separate filter. How cool would that be?

          Show
          Lars Hofhansl added a comment - We could try moving TTL logic into the delete tracker. Would need to check how the stacking works after HBASE-10885 . I think that would be cool. In fact, why not express all the ScanQueryMatcher logic with filters (that's something that our Jesse Yates suggested a long time ago). That would (a) allow users to easily customize the logic and (b) get rid of a whole bunch of custom framework logic. Imagine a FilterList and deals with deletes, TTL, MinVersions, etc, each implemented as a clean, separate filter. How cool would that be?
          Hide
          Andrew Purtell added a comment -

          In fact, why not express all the ScanQueryMatcher logic with filters (that's something that our Jesse Yates suggested a long time ago).

          That would be cool, but perhaps a follow up issue? I.e. move TTL tracking to the SQM as a subtask here, add per cell logic as another, commit both, then complete the suggested larger scope refactor.

          Show
          Andrew Purtell added a comment - In fact, why not express all the ScanQueryMatcher logic with filters (that's something that our Jesse Yates suggested a long time ago). That would be cool, but perhaps a follow up issue? I.e. move TTL tracking to the SQM as a subtask here, add per cell logic as another, commit both, then complete the suggested larger scope refactor.
          Hide
          ramkrishna.s.vasudevan added a comment -

          each implemented as a clean, separate filter

          HBASE-11084 is about that.

          Show
          ramkrishna.s.vasudevan added a comment - each implemented as a clean, separate filter HBASE-11084 is about that.
          Hide
          Lars Hofhansl added a comment -

          HBASE-11084 is about that

          Nice... Missed that.

          That would be cool, but perhaps a follow up issue? I.e. move TTL tracking to the SQM as a subtask here, add per cell logic as another, commit both, then complete the suggested larger scope refactor.

          Let's do that. And use HBASE-11084 for the larger refactor.

          Show
          Lars Hofhansl added a comment - HBASE-11084 is about that Nice... Missed that. That would be cool, but perhaps a follow up issue? I.e. move TTL tracking to the SQM as a subtask here, add per cell logic as another, commit both, then complete the suggested larger scope refactor. Let's do that. And use HBASE-11084 for the larger refactor.
          Hide
          Andrew Purtell added a comment -

          Looks like HBASE-11546 will make .6, let's move this to .7 out of an abundance of caution.

          Show
          Andrew Purtell added a comment - Looks like HBASE-11546 will make .6, let's move this to .7 out of an abundance of caution.
          Hide
          Andrew Purtell added a comment -

          Under active review. Moving out of 0.98.7, should land in 0.98.8

          Show
          Andrew Purtell added a comment - Under active review. Moving out of 0.98.7, should land in 0.98.8
          Hide
          Andrew Purtell added a comment -

          Thanks for the reviews and the 1s. I will commit this to 0.98 just after the 0.98.8 release. Moving to 0.98.9.

          Show
          Andrew Purtell added a comment - Thanks for the reviews and the 1s. I will commit this to 0.98 just after the 0.98.8 release. Moving to 0.98.9.
          Hide
          Andrew Purtell added a comment -

          All subtasks resolved

          Show
          Andrew Purtell added a comment - All subtasks resolved
          Hide
          Andrew Purtell added a comment -

          The work ended up bundled into a combined patch on HBASE-11764, see that issue for patches, but the changes were committed using the id of this issue. A bit weird, sorry about that, subtasks weren't really needed.

          Show
          Andrew Purtell added a comment - The work ended up bundled into a combined patch on HBASE-11764 , see that issue for patches, but the changes were committed using the id of this issue. A bit weird, sorry about that, subtasks weren't really needed.
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-TRUNK #5872 (See https://builds.apache.org/job/HBase-TRUNK/5872/)
          HBASE-10560 Per cell TTLs (apurtell: rev 09cd3d7bfb9f4c4b279a74441d4059da6b8177d2)

          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
          • hbase-shell/src/test/ruby/hbase/table_test.rb
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithTTLs.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTags.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
          • hbase-common/src/main/java/org/apache/hadoop/hbase/TagType.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CellCreator.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/Increment.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/GetClosestRowBeforeTracker.java
          • hbase-shell/src/main/ruby/hbase/table.rb
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/Append.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/Put.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TsvImporterMapper.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-TRUNK #5872 (See https://builds.apache.org/job/HBase-TRUNK/5872/ ) HBASE-10560 Per cell TTLs (apurtell: rev 09cd3d7bfb9f4c4b279a74441d4059da6b8177d2) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-shell/src/test/ruby/hbase/table_test.rb hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithTTLs.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTags.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java hbase-common/src/main/java/org/apache/hadoop/hbase/TagType.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CellCreator.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestScanWildcardColumnTracker.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/Increment.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/GetClosestRowBeforeTracker.java hbase-shell/src/main/ruby/hbase/table.rb hbase-client/src/main/java/org/apache/hadoop/hbase/client/Append.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/Put.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TsvImporterMapper.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-1.0 #546 (See https://builds.apache.org/job/HBase-1.0/546/)
          HBASE-10560 Per cell TTLs (apurtell: rev 004e977ba0009fc8978d842eb65933a62fdf1173)

          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/Increment.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • hbase-common/src/main/java/org/apache/hadoop/hbase/TagType.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithTTLs.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CellCreator.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java
          • hbase-shell/src/test/ruby/hbase/table_test.rb
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/Append.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTags.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TsvImporterMapper.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/Put.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/GetClosestRowBeforeTracker.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
          • hbase-shell/src/main/ruby/hbase/table.rb
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-1.0 #546 (See https://builds.apache.org/job/HBase-1.0/546/ ) HBASE-10560 Per cell TTLs (apurtell: rev 004e977ba0009fc8978d842eb65933a62fdf1173) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/Increment.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-common/src/main/java/org/apache/hadoop/hbase/TagType.java hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithTTLs.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CellCreator.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java hbase-shell/src/test/ruby/hbase/table_test.rb hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/Append.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTags.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TsvImporterMapper.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/Put.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/GetClosestRowBeforeTracker.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java hbase-shell/src/main/ruby/hbase/table.rb
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #689 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/689/)
          HBASE-10560 Per cell TTLs (apurtell: rev 869c5665cd4cc0cd7994933e839154002f682f44)

          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java
          • hbase-common/src/main/java/org/apache/hadoop/hbase/TagType.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TsvImporterMapper.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTags.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
          • hbase-shell/src/test/ruby/hbase/table_test.rb
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/GetClosestRowBeforeTracker.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
          • hbase-shell/src/main/ruby/hbase/table.rb
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CellCreator.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithTTLs.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #689 (See https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/689/ ) HBASE-10560 Per cell TTLs (apurtell: rev 869c5665cd4cc0cd7994933e839154002f682f44) hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java hbase-common/src/main/java/org/apache/hadoop/hbase/TagType.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TsvImporterMapper.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTags.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java hbase-shell/src/test/ruby/hbase/table_test.rb hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/GetClosestRowBeforeTracker.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-shell/src/main/ruby/hbase/table.rb hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CellCreator.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithTTLs.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-0.98 #722 (See https://builds.apache.org/job/HBase-0.98/722/)
          HBASE-10560 Per cell TTLs (apurtell: rev 869c5665cd4cc0cd7994933e839154002f682f44)

          • hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithTTLs.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CellCreator.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
          • hbase-common/src/main/java/org/apache/hadoop/hbase/TagType.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/GetClosestRowBeforeTracker.java
          • hbase-shell/src/main/ruby/hbase/table.rb
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
          • hbase-shell/src/test/ruby/hbase/table_test.rb
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TsvImporterMapper.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTags.java
          • hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java
          • hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
          • hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-0.98 #722 (See https://builds.apache.org/job/HBase-0.98/722/ ) HBASE-10560 Per cell TTLs (apurtell: rev 869c5665cd4cc0cd7994933e839154002f682f44) hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTSVWithTTLs.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/CellCreator.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java hbase-common/src/main/java/org/apache/hadoop/hbase/TagType.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/Mutation.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/GetClosestRowBeforeTracker.java hbase-shell/src/main/ruby/hbase/table.rb hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java hbase-shell/src/test/ruby/hbase/table_test.rb hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TextSortReducer.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TsvImporterMapper.java hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTags.java hbase-client/src/main/java/org/apache/hadoop/hbase/client/Delete.java hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
          Hide
          Enis Soztutar added a comment -

          Closing this issue after 1.0.0 release.

          Show
          Enis Soztutar added a comment - Closing this issue after 1.0.0 release.

            People

            • Assignee:
              Andrew Purtell
              Reporter:
              Andrew Purtell
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development