Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
1.12.0
-
None
-
None
Description
Prerequisites:
1. Create some test table in Hive:
create external table if not exists hive_storage.test (key string, value string) stored as parquet location '/hive_storage/test'; insert into table test values ("key", "value");
2. Hive plugin config:
{ "type": "hive", "enabled": true, "configProps": { "hive.metastore.uris": "thrift://localhost:9083", "fs.default.name": "maprfs:///", "hive.metastore.sasl.enabled": "false" } }
Steps:
1. From the Drill web UI, run the following query:
explain plan for select * from hive.hive_storage.`test`
2. Copy the json part of the plan
3. On the Query page set checkbox to the PHYSICAL
4. Submit the copied plan
Expected result:
Drill should return normal result: "key", "value"
Actual result:
NPE happens:
[Error Id: 8b45c27e-bddd-4552-b7ea-e5af6f40866a on node1:31010] org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: NullPointerException [Error Id: 8b45c27e-bddd-4552-b7ea-e5af6f40866a on node1:31010] at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:633) ~[drill-common-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:761) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.QueryStateProcessor.checkCommonStates(QueryStateProcessor.java:327) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.QueryStateProcessor.planning(QueryStateProcessor.java:223) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:83) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:279) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_161] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_161] Caused by: org.apache.drill.exec.work.foreman.ForemanSetupException: Failure while parsing physical plan. at org.apache.drill.exec.work.foreman.Foreman.parseAndRunPhysicalPlan(Foreman.java:393) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:257) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] ... 3 common frames omitted Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class org.apache.drill.exec.store.hive.HiveScan] value failed (java.lang.NullPointerException): null at [Source: { "head" : { "version" : 1, "generator" : { "type" : "ExplainHandler", "info" : "" }, "type" : "APACHE_DRILL_PHYSICAL", "options" : [ ], "queue" : 0, "hasResourcePlan" : false, "resultMode" : "EXEC" }, "graph" : [ { "pop" : "hive-scan", "@id" : 2, "userName" : "mapr", "hive-table" : { "table" : { "tableName" : "test", "dbName" : "hive_storage", "owner" : "mapr", "createTime" : 1517417959, "lastAccessTime" : 0, "retention" : 0, "sd" : { "location" : "maprfs:/hive_storage/test", "inputFormat" : "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat", "outputFormat" : "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat", "compressed" : false, "numBuckets" : -1, "serDeInfo" : { "name" : null, "serializationLib" : "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe", "parameters" : { "serialization.format" : "1" } }, "sortCols" : [ ], "parameters" : { } }, "partitionKeys" : [ ], "parameters" : { "totalSize" : "0", "EXTERNAL" : "TRUE", "numRows" : "1", "rawDataSize" : "2", "COLUMN_STATS_ACCURATE" : "true", "numFiles" : "0", "transient_lastDdlTime" : "1517418363" }, "viewOriginalText" : null, "viewExpandedText" : null, "tableType" : "EXTERNAL_TABLE", "columnsCache" : { "keys" : [ [ { "name" : "key", "type" : "string", "comment" : null }, { "name" : "value", "type" : "string", "comment" : null } ] ] } }, "partitions" : null }, "columns" : [ "`key`", "`value`" ], "cost" : 0.0 }, { "pop" : "project", "@id" : 1, "exprs" : [ { "ref" : "`key`", "expr" : "`key`" }, { "ref" : "`value`", "expr" : "`value`" } ], "child" : 2, "outputProj" : true, "initialAllocation" : 1000000, "maxAllocation" : 10000000000, "cost" : 1.0 }, { "pop" : "screen", "@id" : 0, "child" : 1, "initialAllocation" : 1000000, "maxAllocation" : 10000000000, "cost" : 1.0 } ] }; line: 1, column: 1420] (through reference chain: org.apache.drill.exec.physical.PhysicalPlan["graph"]->java.util.ArrayList[0]) at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:263) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapAsJsonMappingException(StdValueInstantiator.java:453) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.rewrapCtorProblem(StdValueInstantiator.java:472) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:258) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:135) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:444) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1127) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:298) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId(BeanDeserializerBase.java:1098) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:166) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:135) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:120) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:91) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:142) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:289) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:259) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:26) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:495) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:465) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:384) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1127) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:298) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:133) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1579) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1200) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at org.apache.drill.exec.planner.PhysicalPlanReader.readPhysicalPlan(PhysicalPlanReader.java:96) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman.parseAndRunPhysicalPlan(Foreman.java:390) [drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] ... 4 common frames omitted Caused by: java.lang.NullPointerException: null at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) ~[na:1.8.0_161] at org.apache.drill.exec.store.StoragePluginMap.get(StoragePluginMap.java:106) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.store.StoragePluginRegistryImpl.getPlugin(StoragePluginRegistryImpl.java:273) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.store.hive.HiveScan.<init>(HiveScan.java:88) ~[drill-storage-hive-core-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_161] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_161] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_161] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_161] at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:124) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:256) ~[jackson-databind-2.7.9.1.jar:2.7.9.1] ... 27 common frames omitted 2018-01-31 17:33:27,594 [258e0218-1f39-6078-c31c-e941d3458aa7:foreman] DEBUG o.a.d.e.r.AbstractDisposableUserClientConnection - Result arrived for QueryId: 258e0218-1f39-6078-c31c-e941d3458aa7 with QueryState: FAILED 2018-01-31 17:33:27,599 [qtp226385305-53] ERROR o.a.d.e.server.rest.QueryResources - Query from Web UI Failed org.apache.drill.common.exceptions.UserRemoteException: SYSTEM ERROR: NullPointerException [Error Id: 8b45c27e-bddd-4552-b7ea-e5af6f40866a on node1:31010] (org.apache.drill.exec.work.foreman.ForemanSetupException) Failure while parsing physical plan. org.apache.drill.exec.work.foreman.Foreman.parseAndRunPhysicalPlan():393 org.apache.drill.exec.work.foreman.Foreman.run():257 java.util.concurrent.ThreadPoolExecutor.runWorker():1149 java.util.concurrent.ThreadPoolExecutor$Worker.run():624 java.lang.Thread.run():748 Caused By (com.fasterxml.jackson.databind.JsonMappingException) Instantiation of [simple type, class org.apache.drill.exec.store.hive.HiveScan] value failed (java.lang.NullPointerException): null at [Source: { "head" : { "version" : 1, "generator" : { "type" : "ExplainHandler", "info" : "" }, "type" : "APACHE_DRILL_PHYSICAL", "options" : [ ], "queue" : 0, "hasResourcePlan" : false, "resultMode" : "EXEC" }, "graph" : [ { "pop" : "hive-scan", "@id" : 2, "userName" : "mapr", "hive-table" : { "table" : { "tableName" : "test", "dbName" : "hive_storage", "owner" : "mapr", "createTime" : 1517417959, "lastAccessTime" : 0, "retention" : 0, "sd" : { "location" : "maprfs:/hive_storage/test", "inputFormat" : "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat", "outputFormat" : "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat", "compressed" : false, "numBuckets" : -1, "serDeInfo" : { "name" : null, "serializationLib" : "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe", "parameters" : { "serialization.format" : "1" } }, "sortCols" : [ ], "parameters" : { } }, "partitionKeys" : [ ], "parameters" : { "totalSize" : "0", "EXTERNAL" : "TRUE", "numRows" : "1", "rawDataSize" : "2", "COLUMN_STATS_ACCURATE" : "true", "numFiles" : "0", "transient_lastDdlTime" : "1517418363" }, "viewOriginalText" : null, "viewExpandedText" : null, "tableType" : "EXTERNAL_TABLE", "columnsCache" : { "keys" : [ [ { "name" : "key", "type" : "string", "comment" : null }, { "name" : "value", "type" : "string", "comment" : null } ] ] } }, "partitions" : null }, "columns" : [ "`key`", "`value`" ], "cost" : 0.0 }, { "pop" : "project", "@id" : 1, "exprs" : [ { "ref" : "`key`", "expr" : "`key`" }, { "ref" : "`value`", "expr" : "`value`" } ], "child" : 2, "outputProj" : true, "initialAllocation" : 1000000, "maxAllocation" : 10000000000, "cost" : 1.0 }, { "pop" : "screen", "@id" : 0, "child" : 1, "initialAllocation" : 1000000, "maxAllocation" : 10000000000, "cost" : 1.0 } ] }; line: 1, column: 1420] (through reference chain: org.apache.drill.exec.physical.PhysicalPlan["graph"]->java.util.ArrayList[0]) com.fasterxml.jackson.databind.JsonMappingException.from():263 com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapAsJsonMappingException():453 com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.rewrapCtorProblem():472 com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith():258 com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build():135 com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():444 com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1127 com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject():298 com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1098 com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():166 com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():135 com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():120 com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():91 com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():142 com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize():289 com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize():259 com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize():26 com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize():495 com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping():465 com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():384 com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1127 com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject():298 com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():133 com.fasterxml.jackson.databind.ObjectReader._bindAndClose():1579 com.fasterxml.jackson.databind.ObjectReader.readValue():1200 org.apache.drill.exec.planner.PhysicalPlanReader.readPhysicalPlan():96 org.apache.drill.exec.work.foreman.Foreman.parseAndRunPhysicalPlan():390 org.apache.drill.exec.work.foreman.Foreman.run():257 java.util.concurrent.ThreadPoolExecutor.runWorker():1149 java.util.concurrent.ThreadPoolExecutor$Worker.run():624 java.lang.Thread.run():748 Caused By (java.lang.NullPointerException) null java.util.concurrent.ConcurrentHashMap.get():936 org.apache.drill.exec.store.StoragePluginMap.get():106 org.apache.drill.exec.store.StoragePluginRegistryImpl.getPlugin():273 org.apache.drill.exec.store.hive.HiveScan.<init>():88 sun.reflect.NativeConstructorAccessorImpl.newInstance0():-2 sun.reflect.NativeConstructorAccessorImpl.newInstance():62 sun.reflect.DelegatingConstructorAccessorImpl.newInstance():45 java.lang.reflect.Constructor.newInstance():423 com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call():124 com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith():256 com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build():135 com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():444 com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1127 com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject():298 com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId():1098 com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther():166 com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():135 com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId():120 com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject():91 com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType():142 com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize():289 com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize():259 com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize():26 com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize():495 com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping():465 com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased():384 com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault():1127 com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject():298 com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize():133 com.fasterxml.jackson.databind.ObjectReader._bindAndClose():1579 com.fasterxml.jackson.databind.ObjectReader.readValue():1200 org.apache.drill.exec.planner.PhysicalPlanReader.readPhysicalPlan():96 org.apache.drill.exec.work.foreman.Foreman.parseAndRunPhysicalPlan():390 org.apache.drill.exec.work.foreman.Foreman.run():257 java.util.concurrent.ThreadPoolExecutor.runWorker():1149 java.util.concurrent.ThreadPoolExecutor$Worker.run():624 java.lang.Thread.run():748 at org.apache.drill.exec.rpc.AbstractDisposableUserClientConnection.sendResult(AbstractDisposableUserClientConnection.java:85) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman$ForemanResult.close(Foreman.java:782) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.QueryStateProcessor.checkCommonStates(QueryStateProcessor.java:327) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.QueryStateProcessor.planning(QueryStateProcessor.java:223) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.QueryStateProcessor.moveToState(QueryStateProcessor.java:83) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:279) ~[drill-java-exec-1.13.0-SNAPSHOT.jar:1.13.0-SNAPSHOT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_161] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_161] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_161]
Attachments
Issue Links
- is part of
-
DRILL-6130 Fix NPE during physical plan submission for various storage plugins
-
- Resolved
-