[HIVE-22931] HoS dynamic partitioning fails with blobstore optimizations off

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Assignee: Ádám Szita
• Reporter: Ádám Szita
• Created: 26/Feb/20 09:17
• Updated: 17/Nov/22 08:50
• Resolved: 02/Mar/20 17:02

Description

Reproduction steps:

• Create s3a backed table and normal table.

CREATE TABLE source (
  a string,
  b int,
  c int);
  
CREATE TABLE target (
  a string)
PARTITIONED BY (
  b int,
  c int)
STORED AS parquet
LOCATION
  's3a://somepath';

• Insert values into normal table.

INSERT INTO TABLE source VALUES ("a", "1", "1");

• Do an insert overwrite with dynamic partitions:

set hive.exec.dynamic.partition.mode=nonstrict;
set hive.blobstore.optimizations.enabled=false;
set hive.execution.engine=spark;
INSERT OVERWRITE TABLE target partition (b,c)
SELECT *
FROM source;

This fails only with Spark execution engine + blobstorage optimizations being turned off with:

2020-01-16 15:24:56,064 ERROR hive.ql.metadata.Hive: [load-dynamic-partitions-5]: Exception when loading partition with parameters  partPath=hdfs://nameservice1/tmp/hive/hive/6bcee075-b637-429e-9bf0-a2658355415e/hive_2020-01-16_15-24-01_156_4299941251929377815-4/-mr-10000/.hive-staging_hive_2020-01-16_15-24-01_156_4299941251929377815-4/-ext-10002,  table=email_click_base,  partSpec={b=null, c=null},  replace=true,  listBucketingEnabled=false,  isAcid=false,  hasFollowingStatsTask=trueorg.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Partition spec is incorrect. {companyid=null, eventmonth=null})
        at org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:1666)