Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-16845

INSERT OVERWRITE a table with dynamic partitions on S3 fails with NPE

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.1
    • Fix Version/s: 3.0.0
    • Component/s: None
    • Labels:
      None

      Description

      How to reproduce

      • Create a partitioned table on S3:
        CREATE EXTERNAL TABLE s3table(user_id string COMMENT '', event_name string COMMENT '') PARTITIONED BY (reported_date string, product_id int) LOCATION 's3a://<bucket name>'; 
        
      • Create a temp table:
        create table tmp_table (id string, name string, date string, pid int) row format delimited fields terminated by '\t' lines terminated by '\n' stored as textfile;
        
      • Load the following rows to the tmp table:
        u1	value1	2017-04-10	10000
        u2	value2	2017-04-10	10000
        u3	value3	2017-04-10	10001
        
      • Set the following parameters:
        • hive.exec.dynamic.partition.mode=nonstrict
        • mapreduce.input.fileinputformat.split.maxsize=10
        • hive.blobstore.optimizations.enabled=true
        • hive.blobstore.use.blobstore.as.scratchdir=false
        • hive.merge.mapfiles=true
      • Insert the rows from the temp table into the s3 table:
        INSERT OVERWRITE TABLE s3table
        PARTITION (reported_date, product_id)
        SELECT
          t.id as user_id,
          t.name as event_name,
          t.date as reported_date,
          t.pid as product_id
        FROM tmp_table t;
        

      A NPE will occur with the following stacktrace:

      2017-05-08 21:32:50,607 ERROR org.apache.hive.service.cli.operation.Operation: [HiveServer2-Background-Pool: Thread-184028]: Error running hive query: 
      org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Execution Error, return code -101 from org.apache.hadoop.hive.ql.exec.ConditionalTask. null
      at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:400)
      at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:239)
      at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:88)
      at org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:293)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:415)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
      at org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:306)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.NullPointerException
      at org.apache.hadoop.hive.ql.plan.ConditionalResolverMergeFiles.generateActualTasks(ConditionalResolverMergeFiles.java:290)
      at org.apache.hadoop.hive.ql.plan.ConditionalResolverMergeFiles.getTasks(ConditionalResolverMergeFiles.java:175)
      at org.apache.hadoop.hive.ql.exec.ConditionalTask.execute(ConditionalTask.java:81)
      at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:214)
      at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
      at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1977)
      at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1690)
      at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1422)
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1206)
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1201)
      at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:237)
      ... 11 more 
      

        Attachments

        1. HIVE-16845.1.patch
          33 kB
          Marta Kuczora
        2. HIVE-16845.2.patch
          34 kB
          Marta Kuczora
        3. HIVE-16845.3.patch
          32 kB
          Marta Kuczora
        4. HIVE-16845.4.patch
          32 kB
          Marta Kuczora

          Issue Links

            Activity

              People

              • Assignee:
                kuczoram Marta Kuczora
                Reporter:
                kuczoram Marta Kuczora
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: