Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-21811

Load data into partitioned table throws NPE if DB is enabled for replication.

    XMLWordPrintableJSON

    Details

    • Target Version/s:

      Description

      When load data into a partitioned table with hive.autogather.stats=true and DB is enabled for replication ("repl.source.for" property is set in DB), throws NPE. Here is the call stack.

      0: jdbc:hive2://ctr-e139-1542663976389-126983> LOAD DATA INPATH '/tmp/traffic_data/traffic_data-QUEENS.csv' INTO TABLE traffic_data partition (county='QUEENS');
      INFO  : Loading data to table traffic_database.traffic_data partition (county=QUEENS) from hdfs://ctr-e139-1542663976389-126983-01-000003.hwx.site:8020/tmp/traffic_data/traffic_data-QUEENS.csv
      INFO  : Partition traffic_database.traffic_data{county=QUEENS} stats: [numFiles=1, numRows=0, totalSize=64398392, rawDataSize=0]
      INFO  : [Warning] could not update stats.Failed with exception Unable to alter partition. java.lang.NullPointerException
      org.apache.hadoop.hive.ql.metadata.HiveException: Unable to alter partition. java.lang.NullPointerException
      	at org.apache.hadoop.hive.ql.metadata.Hive.alterPartitions(Hive.java:678)
      	at org.apache.hadoop.hive.ql.exec.StatsTask.aggregateStats(StatsTask.java:261)
      	at org.apache.hadoop.hive.ql.exec.StatsTask.execute(StatsTask.java:122)
      	at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:177)
      	at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:96)
      	at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1777)
      	at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1511)
      	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1308)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1175)
      	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1170)
      	at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:197)
      	at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:76)
      	at org.apache.hive.service.cli.operation.SQLOperation$2$1.run(SQLOperation.java:255)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:422)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
      	at org.apache.hive.service.cli.operation.SQLOperation$2.run(SQLOperation.java:273)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: MetaException(message:java.lang.NullPointerException)
      	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newMetaException(HiveMetaStore.java:6161)
      	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:3908)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
      	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:105)
      	at com.sun.proxy.$Proxy20.alter_partitions_with_environment_context(Unknown Source)
      	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.alter_partitions(HiveMetaStoreClient.java:1485)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:178)
      	at com.sun.proxy.$Proxy21.alter_partitions(Unknown Source)
      	at org.apache.hadoop.hive.ql.metadata.Hive.alterPartitions(Hive.java:676)
      	... 23 more
      Caused by: java.lang.NullPointerException
      	at org.apache.hadoop.hive.metastore.HiveAlterHandler.blockPartitionLocationChangesOnReplSource(HiveAlterHandler.java:729)
      	at org.apache.hadoop.hive.metastore.HiveAlterHandler.alterPartitions(HiveAlterHandler.java:657)
      	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.alter_partitions_with_environment_context(HiveMetaStore.java:3870)
      	... 38 more
      
      No rows affected (2.086 seconds)
      

        Attachments

        1. HIVE-21811.01.patch
          5 kB
          Sankar Hariappan

          Issue Links

            Activity

              People

              • Assignee:
                sankarh Sankar Hariappan
                Reporter:
                sankarh Sankar Hariappan
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m