[HIVE-18589] java.io.IOException: Not enough history available

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Transactions
• Target Version/s: 3.0.0
• Assignee: Eugene Koifman
• Reporter: Eugene Koifman
• Created: 31/Jan/18 04:37
• Updated: 17/Aug/18 01:03
• Resolved: 02/Feb/18 23:41

Description

txnid:17 starts reading T2/P1

txnid:20 does insert overwrite T1/P1, creates base_20, commits.  txnid:17 still running

txnid:21 stars reading T1/P1.  It's ValidTxnList will txnid:17 as open.

before Insert overwrite was supported, only the compactor could produce base_20 by running major compaction.  Major compaction erases history and so a reader with txnid:17 open, can't use base_20.

Normally, the Cleaner is smart enough to not clean pre-compaction files if it's possible that there is a reader that requires them.  There is a safety check that creates "Not enough history.." error if it finds that the current reader can't properly execute based on the files available.

 

with the introduction of IOW on acid tables, there is another way to produce a base.  The difference is that here, the base has no history by definition and so the same check is not needed but is triggered in the scenario above.

 

 

Issue Links

• is related to: HIVE-20410 aborted Insert Overwrite on transactional table causes "Not enough history available for..." error
• is required by: HIVE-17457 IOW Acid Insert Overwrite when the transaction fails
• relates to: HIVE-17856 MM tables - IOW is not ACID compliant
• relates to: HIVE-14988 Support INSERT OVERWRITE into a partition on transactional tables
• relates to: HIVE-17361 Support LOAD DATA for transactional tables