[HIVE-13646] make hive.optimize.sort.dynamic.partition compatible with ACID tables

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 1.3.0, 2.1.0
• Component/s: Transactions
• Target Version/s: 1.3.0, 2.1.0
• Assignee: Eugene Koifman
• Reporter: Eugene Koifman
• Created: 28/Apr/16 20:34
• Updated: 21/Jun/16 15:10
• Resolved: 04/May/16 00:12

Description

HIVE-8875 disabled hive.optimize.sort.dynamic.partition for ACID queries.
dynamic inserts are common in ACID and this leaves users with few options if they are seeing OutOfMemory errors due to too many writers.

hive.optimize.sort.dynamic.partition sorts data by partition col/bucket col/sort col to ensure each reducer only needs 1 writer.

Acid requires data in each bucket file to be sorted by ROW__ID and thus doesn't allow end user to determine sorting.

So we should be able to support hive.optimize.sort.dynamic.partition with
sort on partition col/bucket col/ROW__ID

Issue Links

• blocks: HIVE-11550 ACID queries pollute HiveConf
• is blocked by: HIVE-13773 Stats state is not captured correctly in dynpart_sort_optimization_acid.q