[HIVE-16845] INSERT OVERWRITE a table with dynamic partitions on S3 fails with NPE

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.1
• Fix Version/s: 3.0.0
• Assignee: Marta Kuczora
• Reporter: Marta Kuczora
• Created: 07/Jun/17 17:07
• Updated: 23/May/18 00:00
• Resolved: 03/Aug/17 10:17

Description

How to reproduce

• Create a partitioned table on S3:

  CREATE EXTERNAL TABLE s3table(user_id string COMMENT '', event_name string COMMENT '') PARTITIONED BY (reported_date string, product_id int) LOCATION 's3a://<bucket name>'; 
  

• Create a temp table:

  create table tmp_table (id string, name string, date string, pid int) row format delimited fields terminated by '\t' lines terminated by '\n' stored as textfile;
  

• Load the following rows to the tmp table:

  u1	value1	2017-04-10	10000
  u2	value2	2017-04-10	10000
  u3	value3	2017-04-10	10001
  

• Set the following parameters:
  • hive.exec.dynamic.partition.mode=nonstrict
  • mapreduce.input.fileinputformat.split.maxsize=10
  • hive.blobstore.optimizations.enabled=true
  • hive.blobstore.use.blobstore.as.scratchdir=false
  • hive.merge.mapfiles=true
• Insert the rows from the temp table into the s3 table:

  INSERT OVERWRITE TABLE s3table
  PARTITION (reported_date, product_id)
  SELECT
    t.id as user_id,
    t.name as event_name,
    t.date as reported_date,
    t.pid as product_id
  FROM tmp_table t;
  

A NPE will occur with the following stacktrace:

2017-05-08 21:32:50,607 ERROR org.apache.hive.service.cli.operation.Operation: [HiveServer2-Background-Pool: Thread-184028]: Error running hive query: 
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.ConditionalTask. null
at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:400)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:239)
at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
at org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
at org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.plan.ConditionalResolverMergeFiles.generateActualTasks(ConditionalResolverMergeFiles.java:290)
at org.apache.hadoop.hive.ql.plan.ConditionalResolverMergeFiles.getTasks(ConditionalResolverMergeFiles.java:175)
at org.apache.hadoop.hive.ql.exec.ConditionalTask.execute(ConditionalTask.java:81)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1977)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1690)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1422)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1206)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1201)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
... 11 more 

Issue Links

• links to: Review