[TEPHRA-35] Prune invalid transaction set once all data for a given invalid transaction has been dropped

• Type: New Feature
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.11.0-incubating
• Assignee: Poorna Chandra
• Reporter: Gary Helmling
• Created: 28/Aug/14 22:04
• Updated: 11/Sep/17 05:06
• Resolved: 07/Mar/17 17:06

Description

In addition to dropping the data from invalid transactions we need to be able to prune the invalid set of any transactions where data cleanup has been completely performed. Without this, the invalid set will grow indefinitely and become a greater and greater cost to in-progress transactions over time.

To do this correctly, the TransactionDataJanitor coprocessor will need to maintain some bookkeeping for the transaction data that it removes, so that the transaction manager can reason about when all of a given transaction's data has been removed. Only at this point can the transaction manager safely drop the transaction ID from the invalid set.

One approach would be for the TransactionDataJanitor to update a table marking when a major compaction was performed on a region and what transaction IDs were filtered out. Once all regions in a table containing the transaction data have been compacted, we can remove the filtered out transaction IDs from the invalid set. However, this will need to cope with changing region names due to splits, etc.

Issue Links

• is duplicated by: PHOENIX-2993 Tephra: Prune invalid transaction set once all data for a given invalid transaction has been dropped
• is related to: TEPHRA-254 Handle transient tables that get recreated multiple times during pruning
• is related to: TEPHRA-244 Invalid tx pruning does not handle deletion of tables well
• is related to: TEPHRA-205 Load transaction pruning plugins in separate classloaders
• relates to: PHOENIX-3037 Setup proper security context in compaction/split coprocessor hooks
• relates to: TEPHRA-196 Invalid transaction list pruning should work with stripe compactions
• links to: CDAP-3383 Automate pruning of invalid transaction list after data for the invalid transaction has been dropped
• links to: GitHub Pull Request #19

Sub-Tasks

• Compute global prune upper bound using compaction state of every region: Resolved
• Enforce maximum duration a transaction can be used for data writes: Resolved
• Schedule Transaction Pruning periodically: Resolved
• Document invalid transaction list pruning: Open
• Port invalid list pruning changes to other compat modules: Resolved
• Invalid Transaction List Pruning will not proceed if there are empty transactional tables: Resolved
• Setup proper security context in co-processor for compaction and flushes: Resolved
• Handle delay between transaction max lifetime check and data writes while pruning: Resolved