[HIVE-20723] Allow per table specification of compaction yarn queue

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Transactions
• Target Version/s: 4.0.0
• Assignee: Saurabh Seth
• Reporter: Eugene Koifman
• Created: 10/Oct/18 18:56
• Updated: 17/Nov/22 08:55
• Resolved: 14/Oct/18 00:11

Description

Currently compactions of full CRUD transactional tables are Map-Reduce jobs submitted to a yarn queue defined by hive.compactor.job.queue property.

If would be useful to be able to override this on per table basis by putting it into table properties so that compactions for different tables can use different queues.

 

There is already ability to override other compaction related configs via table props, though this will need additional handling to set the queue name CompactorMr.createBaseJobConf

https://cwiki.apache.org/confluence/display/Hive/Hive+Transactions#HiveTransactions-TableProperties

 

See CopactorMR.COMPACTOR_PREFIX and Initiator.COMPACTORTHRESHOLD_PREFIX

 

 

 

Issue Links

• causes: HIVE-25947 Compactor job queue cannot be set per table via compactor.mapred.job.queue.name
• is related to: HIVE-20874 Add ability to to run high priority compaction