[HIVE-23442] ACID major compaction doesn't read base directory correctly if it was written by insert overwrite

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0-alpha-1
• Assignee: Marta Kuczora
• Reporter: Marta Kuczora
• Created: 11/May/20 18:20
• Updated: 17/Nov/22 08:48
• Resolved: 12/May/20 11:55

Description

Steps to reproduce:

SET hive.acid.direct.insert.enabled=true;

CREATE EXTERNAL TABLE test_comp_txt(a int, b int, c int) STORED AS TEXTFILE;
INSERT INTO test_comp_txt values (1, 1, 1), (2, 2, 2), (3, 3, 3), (4, 4, 4);

CREATE TABLE test_comp(a int, b int, c int) STORED AS ORC TBLPROPERTIES('transactional'='true');
INSERT OVERWRITE TABLE test_comp SELECT * FROM test_comp_txt;

UPDATE test_comp SET b=55, c=66 WHERE a=2;
DELETE FROM test_comp WHERE a=4;
UPDATE test_comp SET b=77 WHERE a=1;

SELECT * FROM test_comp;
3	3	3
2	55	66
1	77	1

ALTER TABLE test_comp COMPACT 'MAJOR';

SELECT * FROM test_comp;
2	55	66
1	77	1

This issue only occurs if the base directory was created with an insert overwrite command and the hive.acid.direct.insert.enabled parameter was true. This issue doesn't affect the query based compaction.

Issue Links

• relates to: HIVE-21164 ACID: explore how we can avoid a move step during inserts/compaction