[IMPALA-10996] Flaky dataloading: too many dynamic partitions per node in Hive

• Type: Bug
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Labels: flaky-build
• Epic Color: ghx-label-1
• Assignee: Unassigned
• Reporter: Csaba Ringhofer
• Created: 29/Oct/21 13:55
• Updated: 29/Oct/21 13:55

Description

The following query led to failure:

NSERT OVERWRITE TABLE tpcds_seq_snap.store_sales PARTITION (ss_sold_date_sk)
SELECT ss_sold_time_sk,
ss_item_sk,
ss_customer_sk,
ss_cdemo_sk,
ss_hdemo_sk,
ss_addr_sk,
ss_store_sk,
ss_promo_sk,
ss_ticket_number,
ss_quantity,
ss_wholesale_cost,
ss_list_price,
ss_sales_price,
ss_ext_discount_amt,
ss_ext_sales_price,
ss_ext_wholesale_cost,
ss_ext_list_price,
ss_ext_tax,
ss_coupon_amt,
ss_net_paid,
ss_net_paid_inc_tax,
ss_net_profit,
ss_sold_date_sk
FROM tpcds.store_sales
WHERE ss_sold_date_sk < 2451272

Error:

], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) : attempt_1635481802246_0012_19_00_000001_1:java.lang.RuntimeException: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveFatalException: [Error 20004]: Fatal error occurred when node tried to create too many dynamic partitions. The maximum number of dynamic partitions is controlled by hive.exec.max.dynamic.partitions and hive.exec.max.dynamic.partitions.pernode. Maximum was set to 100 partitions per node, number of dynamic partitions on this node: 101