Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
None
Description
Reproduction steps:
- Create s3a backed table and normal table.
CREATE TABLE source ( a string, b int, c int); CREATE TABLE target ( a string) PARTITIONED BY ( b int, c int) STORED AS parquet LOCATION 's3a://somepath';
- Insert values into normal table.
INSERT INTO TABLE source VALUES ("a", "1", "1");
- Do an insert overwrite with dynamic partitions:
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.blobstore.optimizations.enabled=false;
set hive.execution.engine=spark;
INSERT OVERWRITE TABLE target partition (b,c)
SELECT *
FROM source;
This fails only with Spark execution engine + blobstorage optimizations being turned off with:
2020-01-16 15:24:56,064 ERROR hive.ql.metadata.Hive: [load-dynamic-partitions-5]: Exception when loading partition with parameters partPath=hdfs://nameservice1/tmp/hive/hive/6bcee075-b637-429e-9bf0-a2658355415e/hive_2020-01-16_15-24-01_156_4299941251929377815-4/-mr-10000/.hive-staging_hive_2020-01-16_15-24-01_156_4299941251929377815-4/-ext-10002, table=email_click_base, partSpec={b=null, c=null}, replace=true, listBucketingEnabled=false, isAcid=false, hasFollowingStatsTask=trueorg.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Partition spec is incorrect. {companyid=null, eventmonth=null})
at org.apache.hadoop.hive.ql.metadata.Hive.loadPartitionInternal(Hive.java:1666)