Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-1324

Druid metadata query throws exception if there are non-standard aggregators

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: 1.9.0
    • Fix Version/s: 1.9.0
    • Component/s: druid
    • Labels:
      None

      Description

      If the approximate histogram aggregator is used the druid metadata query fails as the aggregator json contains unexpected fields.

      Quick patch which just ignores the fields and the type: https://github.com/remerge/calcite/commit/6752c302eb95a294380891c6de8dca6349fe1ed6

      com.google.common.util.concurrent.UncheckedExecutionException: java.lang.RuntimeException: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "resolution" (class org.apache.calcite.adapter.druid.DruidConnectionImpl$JsonAggregator), not marked as ignorable (3 known properties: "type", "name", "fieldName"])
       at [Source: sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@39bebb16; line: 208, column: 24] (through reference chain: java.util.ArrayList[0]->org.apache.calcite.adapter.druid.JsonSegmentMetadata["aggregators"]->java.util.LinkedHashMap["bid_price_hist_eur"]->org.apache.calcite.adapter.druid.JsonAggregator["resolution"])
      	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)
      	at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
      	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
      	at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
      	at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
      	at com.google.common.cache.LocalCache$LocalLoadingCache.apply(LocalCache.java:4848)
      	at com.google.common.collect.Maps$AsMapView.get(Maps.java:793)
      	at org.apache.calcite.schema.impl.AbstractSchema.getTable(AbstractSchema.java:91)
      	at org.apache.calcite.jdbc.CachingCalciteSchema.getImplicitTable(CachingCalciteSchema.java:127)
      	at org.apache.calcite.jdbc.CalciteSchema.getTable(CalciteSchema.java:255)
      	at org.apache.calcite.jdbc.CalciteMetaImpl$12.apply(CalciteMetaImpl.java:417)
      	at org.apache.calcite.jdbc.CalciteMetaImpl$12.apply(CalciteMetaImpl.java:415)
      	at org.apache.calcite.linq4j.EnumerableDefaults$15$1.current(EnumerableDefaults.java:1893)
      	at org.apache.calcite.linq4j.Linq4j$CompositeEnumerable$1.current(Linq4j.java:498)
      	at org.apache.calcite.linq4j.EnumerableDefaults$33.moveNext(EnumerableDefaults.java:2805)
      	at org.apache.calcite.linq4j.EnumerableDefaults$17$1.moveNext(EnumerableDefaults.java:1970)
      	at org.apache.calcite.linq4j.EnumerableDefaults$33.moveNext(EnumerableDefaults.java:2804)
      	at org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.next(Linq4j.java:673)
      	at org.apache.calcite.avatica.util.IteratorCursor.next(IteratorCursor.java:46)
      	at org.apache.calcite.avatica.AvaticaResultSet.next(AvaticaResultSet.java:218)
      	at org.apache.calcite.jdbc.CalciteResultSet.next(CalciteResultSet.java:86)
      	at sqlline.IncrementalRows.hasNext(IncrementalRows.java:62)
      	at sqlline.TableOutputFormat.print(TableOutputFormat.java:33)
      	at sqlline.SqlLine.print(SqlLine.java:1652)
      	at sqlline.Commands.metadata(Commands.java:199)
      	at sqlline.Commands.tables(Commands.java:332)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at sqlline.ReflectiveCommandHandler.execute(ReflectiveCommandHandler.java:36)
      	at sqlline.SqlLine.dispatch(SqlLine.java:803)
      	at sqlline.SqlLine.begin(SqlLine.java:681)
      	at sqlline.SqlLine.start(SqlLine.java:398)
      	at sqlline.SqlLine.main(SqlLine.java:292)
      Caused by: java.lang.RuntimeException: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "resolution" (class org.apache.calcite.adapter.druid.DruidConnectionImpl$JsonAggregator), not marked as ignorable (3 known properties: "type", "name", "fieldName"])
       at [Source: sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@39bebb16; line: 208, column: 24] (through reference chain: java.util.ArrayList[0]->org.apache.calcite.adapter.druid.JsonSegmentMetadata["aggregators"]->java.util.LinkedHashMap["bid_price_hist_eur"]->org.apache.calcite.adapter.druid.JsonAggregator["resolution"])
      	at com.google.common.base.Throwables.propagate(Throwables.java:160)
      	at org.apache.calcite.adapter.druid.DruidConnectionImpl.metadata(DruidConnectionImpl.java:375)
      	at org.apache.calcite.adapter.druid.DruidSchema$1.load(DruidSchema.java:72)
      	at org.apache.calcite.adapter.druid.DruidSchema$1.load(DruidSchema.java:68)
      	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
      	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
      	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
      	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
      	... 34 more
      Caused by: com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "resolution" (class org.apache.calcite.adapter.druid.DruidConnectionImpl$JsonAggregator), not marked as ignorable (3 known properties: "type", "name", "fieldName"])
       at [Source: sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@39bebb16; line: 208, column: 24] (through reference chain: java.util.ArrayList[0]->org.apache.calcite.adapter.druid.JsonSegmentMetadata["aggregators"]->java.util.LinkedHashMap["bid_price_hist_eur"]->org.apache.calcite.adapter.druid.JsonAggregator["resolution"])
      	at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:51)
      	at com.fasterxml.jackson.databind.DeserializationContext.reportUnknownProperty(DeserializationContext.java:839)
      	at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:1045)
      	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1352)
      	at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1330)
      	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:264)
      	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:125)
      	at com.fasterxml.jackson.databind.deser.std.MapDeserializer._readAndBindStringMap(MapDeserializer.java:495)
      	at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:341)
      	at com.fasterxml.jackson.databind.deser.std.MapDeserializer.deserialize(MapDeserializer.java:26)
      	at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:520)
      	at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:101)
      	at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:258)
      	at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:125)
      	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:245)
      	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:217)
      	at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:25)
      	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3736)
      	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2810)
      	at org.apache.calcite.adapter.druid.DruidConnectionImpl.metadata(DruidConnectionImpl.java:360)
      	... 40 more
      

        Activity

        Hide
        julianhyde Julian Hyde added a comment -

        Good find. (Yes, I kind of knew this was going to be a problem.) That patch looks good also. Can you please create a pull request to https://github.com/apache/calcite; this creates a record of your consent to contribute to the Apache project.

        Show
        julianhyde Julian Hyde added a comment - Good find. (Yes, I kind of knew this was going to be a problem.) That patch looks good also. Can you please create a pull request to https://github.com/apache/calcite ; this creates a record of your consent to contribute to the Apache project.
        Hide
        martin.karlsch Martin Karlsch added a comment -

        Done: https://github.com/apache/calcite/pull/259. For now it is just a workaround.

        Show
        martin.karlsch Martin Karlsch added a comment - Done: https://github.com/apache/calcite/pull/259 . For now it is just a workaround.
        Hide
        julianhyde Julian Hyde added a comment - - edited

        Fixed in http://git-wip-us.apache.org/repos/asf/calcite/commit/6002cd28. Thanks for the PR, Martin Karlsch!

        The fix silently ignores non-standard aggregators.

        Show
        julianhyde Julian Hyde added a comment - - edited Fixed in http://git-wip-us.apache.org/repos/asf/calcite/commit/6002cd28 . Thanks for the PR, Martin Karlsch ! The fix silently ignores non-standard aggregators.
        Hide
        jcamachorodriguez Jesus Camacho Rodriguez added a comment -

        Resolved in release 1.9.0 (2016-09-22)

        Show
        jcamachorodriguez Jesus Camacho Rodriguez added a comment - Resolved in release 1.9.0 (2016-09-22)

          People

          • Assignee:
            julianhyde Julian Hyde
            Reporter:
            martin.karlsch Martin Karlsch
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development