Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-5645

BaseScannerRegionObserver should prevent compaction from purging very recently deleted cells

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 4.14.4, 4.16.0
    • None
    • None

    Description

      Phoenix's SCN feature has some problems, because HBase major compaction can remove Cells that have been deleted or whose TTL or max versions has caused them to be expired.

      For example, IndexTool rebuilds and index scrutiny can both give strange, incorrect results if a major compaction occurs in the middle of their run. In the rebuild case, it's because we're rewriting "history" on the index at the same time that compaction is rewriting "history" by purging deleted and expired cells.

      Create a new configuration property called "max lookback age", which declares that no data written more recently than the max lookback age will be compacted away. The max lookback age must be smaller than the TTL, and it should not be legal for a user to look back further in the past than the table's TTL.

      Max lookback age by default will not be set, and the current behavior will be preserved. But if max lookback age is set, it will be enforced by the BaseScannerRegionObserver for all tables.

      In the future, this should be contributed as a general feature to HBase for arbitrary tables. See HBASE-23602.

      Attachments

        1. PHOENIX-5645-4.x-HBase-1.5.patch
          17 kB
          Geoffrey Jacoby
        2. PHOENIX-5645-4.x-HBase-1.5-v2.patch
          63 kB
          Geoffrey Jacoby
        3. PHOENIX-5645-4.x-HBase-1.5.v3.patch
          63 kB
          Geoffrey Jacoby
        4. PHOENIX-5645-addendum-4.x-HBase-1.5.patch
          26 kB
          Geoffrey Jacoby
        5. PHOENIX-5645-4.14-HBase-1.4.patch
          66 kB
          Geoffrey Jacoby
        6. PHOENIX-5645-4.14-HBase-1.3.v2.patch
          66 kB
          Geoffrey Jacoby

        Activity

          People

            gjacoby Geoffrey Jacoby
            gjacoby Geoffrey Jacoby
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 9h 40m
                9h 40m