Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-6111

NullPointerException with Kafka Storage Plugin

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.12.0
    • Fix Version/s: 1.13.0
    • Component/s: Storage - Other
    • Labels:
      None

      Description

      I'm unable to query using the kafka storage plugin; queries are failing with a NPE which seems like a json typo:

      org.apache.drill.common.exceptions.UserException: SYSTEM ERROR: NullPointerException
      Fragment 1:2
      [Error Id: 49d5f72f-0187-480b-8b29-6eeeb5adc88f on 10.80.53.16:31820]
      at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:586) ~[drill-common-1.12.0.jar:1.12.0]
      at org.apache.drill.exec.work.fragment.FragmentExecutor.sendFinalState(FragmentExecutor.java:298) [drill-java-exec-1.12.0.jar:1.12.0]
      at org.apache.drill.exec.work.fragment.FragmentExecutor.cleanup(FragmentExecutor.java:160) [drill-java-exec-1.12.0.jar:1.12.0]
      at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:267) [drill-java-exec-1.12.0.jar:1.12.0]
      at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38) [drill-common-1.12.0.jar:1.12.0]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
      at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
      Caused by: com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class org.apache.drill.exec.store.kafka.KafkaSubScan] value failed (java.lang.NullPointerException): null
      at [Source: {
      "pop" : "single-sender",
      "@id" : 0,
      "receiver-major-fragment" : 0,
      "receiver-minor-fragment" : 0,
      "child" : {
      "pop" : "selection-vector-remover",
      "@id" : 1,
      "child" : {
      "pop" : "limit",
      "@id" : 2,
      "child" : {
      "pop" : "kafka-partition-scan",
      "@id" : 3,
      "userName" : "",
      "columns" : [ "`*`" ],
      "partitionSubScanSpecList" : [ {
      "topicName" : "ingest-prime",
      "partitionId" : 5,
      "startOffset" : 8824294,
      "endOffset" : 8874172
      }, {
      "topicName" : "ingest-prime",
      "partitionId" : 1,
      "startOffset" : 8826346,
      "endOffset" : 8874623
      }, {
      "topicName" : "ingest-prime",
      "partitionId" : 6,
      "startOffset" : 8824744,
      "endOffset" : 8874617
      } ],
      "initialAllocation" : 1000000,
      "maxAllocation" : 10000000000,
      "KafkaStoragePluginConfig" : {
      "type" : "kafka",
      "kafkaConsumerProps" : {
      "key.deserializer" : "org.apache.kafka.common.serialization.ByteArrayDeserializer",
      "auto.offset.reset" : "earliest",
      "bootstrap.servers" : "kafkas.dev3.master.us-west-2.prod.aws.intellify.io:9092",
      "enable.auto.commit" : "true",
      "group.id" : "drill-query-consumer-1",
      "value.deserializer" : "org.apache.kafka.common.serialization.ByteArrayDeserializer",
      "session.timeout.ms" : "30000"
      },
      "enabled" : true
      },
      "cost" : 0.0
      },
      "first" : 0,
      "last" : 2,
      "initialAllocation" : 1000000,
      "maxAllocation" : 10000000000,
      "cost" : 2.0
      },
      "initialAllocation" : 1000000,
      "maxAllocation" : 10000000000,
      "cost" : 2.0
      },
      "destination" : "CgsxMC44MC41My4xNhDM+AEYzfgBIM74ATIGMS4xMi4wOAA=",
      "initialAllocation" : 1000000,
      "maxAllocation" : 10000000000,
      "cost" : 2.0
      }; line: 49, column: 7] (through reference chain: org.apache.drill.exec.physical.config.SingleSender["child"]->org.apache.drill.exec.physical.config.SelectionVectorRemover["child"]->org.apache.drill.exec.physical.config.Limit["child"])
      at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:263) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.wrapAsJsonMappingException(StdValueInstantiator.java:453) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.rewrapCtorProblem(StdValueInstantiator.java:472) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:258) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.impl.PropertyBasedCreator.build(PropertyBasedCreator.java:135) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:444) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1127) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:298) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId(BeanDeserializerBase.java:1098) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:166) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:135) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:120) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:91) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:142) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:493) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:465) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:384) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1127) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:298) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId(BeanDeserializerBase.java:1098) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:166) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:135) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:120) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:91) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:142) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:493) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:465) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:384) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1127) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:298) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId(BeanDeserializerBase.java:1098) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:166) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:135) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:120) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:91) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:142) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:493) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeWithErrorWrapping(BeanDeserializer.java:465) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeUsingPropertyBased(BeanDeserializer.java:384) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromObjectUsingNonDefault(BeanDeserializerBase.java:1127) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserializeFromObject(BeanDeserializer.java:298) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeWithObjectId(BeanDeserializerBase.java:1098) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:166) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:135) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer._deserializeTypedForId(AsPropertyTypeDeserializer.java:120) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.jsontype.impl.AsPropertyTypeDeserializer.deserializeTypedFromObject(AsPropertyTypeDeserializer.java:91) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.AbstractDeserializer.deserializeWithType(AbstractDeserializer.java:142) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.impl.TypeWrappedDeserializer.deserialize(TypeWrappedDeserializer.java:63) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1579) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1200) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at org.apache.drill.exec.planner.PhysicalPlanReader.readFragmentRoot(PhysicalPlanReader.java:101) ~[drill-java-exec-1.12.0.jar:1.12.0]
      at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:204) [drill-java-exec-1.12.0.jar:1.12.0]
      ... 4 common frames omitted
      Caused by: java.lang.NullPointerException: null
      at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:210) ~[guava-18.0.jar:na]
      at com.google.common.cache.LocalCache.get(LocalCache.java:3936) ~[guava-18.0.jar:na]
      at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) ~[guava-18.0.jar:na]
      at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) ~[guava-18.0.jar:na]
      at org.apache.drill.exec.store.StoragePluginRegistryImpl.getPlugin(StoragePluginRegistryImpl.java:311) ~[drill-java-exec-1.12.0.jar:1.12.0]
      at org.apache.drill.exec.store.kafka.KafkaSubScan.<init>(KafkaSubScan.java:62) ~[drill-storage-kafka-1.12.0.jar:1.12.0]
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.8.0_131]
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[na:1.8.0_131]
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.8.0_131]
      at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[na:1.8.0_131]
      at com.fasterxml.jackson.databind.introspect.AnnotatedConstructor.call(AnnotatedConstructor.java:124) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      at com.fasterxml.jackson.databind.deser.std.StdValueInstantiator.createFromObjectWith(StdValueInstantiator.java:256) ~[jackson-databind-2.7.9.1.jar:2.7.9.1]
      ... 52 common frames omitted

      Looking at the source, it seems like it's expecting the json field to be named

      @JsonProperty("kafkaStoragePluginConfig")

      (note the lower-cased 'k' at the start), but in the snippet in the error message it starts with a capital 'K'.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                arina Arina Ielchiieva
                Reporter:
                jstehler Jared Stehler
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: