[HIVE-27598] Enhance alter table compact to work for partitioned tables without specifying any partition

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-beta-1
• Component/s: Hive
• Labels: pull-request-available
• Assignee: Taraka Rama Rao Lethavadla
• Reporter: Taraka Rama Rao Lethavadla
• Created: 11/Aug/23 08:39
• Updated: 19/Mar/24 10:16
• Resolved: 31/Oct/23 11:19

Description

current implementation: the below command will throw error
 

alter table PARTITIONED_T compact 'minor';

for a partitioned table

You must specify a partition to compact for partitioned tables

Problem: One of the problem could be ,let's say a table contains 1000s of partitions and for some issue like files permission automatic compaction will mark all or few partitions as failed and never pick for compaction. 

After we see performance getting degraded, we find out that there is a problem in compaction due to some issue.

Now to solve it, we have two approaches, adjust failure thresholds and restart HMS to pick the changes which is not going to work for someone who cannot restart HMS

And the second approach is to execute alter table mentioning each partition

Solution:

In this Jira we are planning to allow alter table command for a partitioned table without specifying any partition name.

Proposed solution is to list all the partitions and find partitions that are eligible to get compacted and put an entry for it in compaction queue

Issue Links

• causes: HIVE-27848 Refactor Initiator hierarchy into CompactorUtil and fix failure in TestCrudCompactorOnTez
• links to: GitHub Pull Request #4577