[HIVE-22538] RS deduplication does not always enforce hive.optimize.reducededuplication.min.reducer

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: Physical Optimizer
• Labels: pull-request-available
• Assignee: Krisztian Kasa
• Reporter: Jesús Camacho Rodríguez
• Created: 26/Nov/19 00:58
• Updated: 27/Feb/24 22:23
• Resolved: 28/Jan/20 21:00

Description

For transactional tables, that property might be overriden to 1, which can lead to merging final aggregation into a single stage (hence leading to performance degradation). For instance, when autogather column stats is enabled, this can happen for the following query:

set hive.support.concurrency=true;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;

EXPLAIN
CREATE TABLE x STORED AS ORC TBLPROPERTIES('transactional'='true') AS
SELECT * FROM SRC x CLUSTER BY x.key;

Issue Links

• fixes: HIVE-22549 RS deduplication should not merge final aggregation without keys
• incorporates: HIVE-23889 Empty bucket files are inserted with invalid schema after HIVE-21784
• is related to: HIVE-25611 OOM when running MERGE query on wide transactional table with many buckets
• relates to: HIVE-25071 Number of reducers limited to fixed 1 when updating/deleting
• links to: GitHub Pull Request #855
• links to: GitHub Pull Request #877