Details
Description
This will enable pushing down of filter with time extraction to druid.
Only day/month/year is supported.
Issue Links
- depends upon
-
CALCITE-1730
Upgrade Druid to 0.9.2
-
- Closed
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
Jesus Camacho Rodriguez and Julian Hyde can this make it by the 1.12 release ?
I don't think so. It's a major change and you haven't written any tests.
Also, please undo your whitespace/formatting changes, and make sure you adhere to the project's coding style (e.g. continuation indent 4, close parens on the same line).
is there a formatting template i can use with intellij ?
No, but feel free to create one consistent with the house style.
Ok is there a list of rules ?
Julian Hyde i have added tests, any link to the project code style ?
The code style isn’t documented (contributions welcome!) but it’s pretty clear looking at the existing code. The goal is that you can’t tell which developer wrote the code.
Checkstyle, which gets run as part of “mvn validate” will point out certain classes of errors. See src/main/config/checkstyle/.
slim bouguerra, I was reviewing your PR. I run the Druid testsuite and testFilterTimestamp is failing.
Failed tests:
DruidAdapterIT.testFilterTimestamp:1421
Expected: a string containing "EnumerableInterpreter\n BindableAggregate(group=[{}], C=[COUNT()])\n BindableFilter(condition=[AND(>=(/INT(Reinterpret($0), 86400000), 1997-01-01), <(/INT(Reinterpret($0), 86400000), 1998-01-01), OR(AND(>=(/INT(Reinterpret($0), 86400000), 1997-04-01), <(/INT(Reinterpret($0), 86400000), 1997-05-01)), AND(>=(/INT(Reinterpret($0), 86400000), 1997-06-01), <(/INT(Reinterpret($0), 86400000), 1997-07-01))))])\n DruidQuery(table=[[foodmart, foodmart]], intervals=[[1900-01-09T00:00:00.000/2992-01-10T00:00:00.000]], projects=[[$0]])"
but: was "PLAN=EnumerableInterpreter
DruidQuery(table=[[foodmart, foodmart]], intervals=[[]], filter=[=(EXTRACT_DATE(FLAG(YEAR), /INT(Reinterpret($0), 86400000)), 1997)], groups=[{}], aggs=[[COUNT()]])
Change in the plan is incorrect. Observe that intervals is empty and filter condition does not contain all the information it should (it only contains the filter by year).
I added an additional test without extract year in the Filter condition and the result is similar: intervals get lost.
We need to take a look at those errors before we can push this fix.
Jesus Camacho Rodriguez thanks will look into that.
Jesus Camacho Rodriguez please checkout the new fix.
FYI the plan is good but the extract is failing on druid i am guess it needs druid 0.9.2 at least.
as per the logs from the broker there is an exception due to the extraction function call. This issue is fixed in 0.9.2
2017-03-29T03:37:57,112 INFO [qtp448782465-23[timeseries_foodmart_1910fc54-91e6-4fd1-96eb-2147775f5027]] com.metamx.emitter.core.LoggingEmitter - Event [{"feed":"metrics","timestamp":"2017-03-29T03:37:57.112Z","service":"druid/broker","host":"ubuntucalcite:8082","metric":"query/time","value":4,"context":"{\"queryId\":\"1910fc54-91e6-4fd1-96eb-2147775f5027\",\"sk
ipEmptyBuckets\":true,\"timeout\":300000}","dataSource":"foodmart","duration":"PT34460294400S","hasFilters":"true","id":"1910fc54-91e6-4fd1-96eb-2147775f5027","interval":["1900-01-09T00:00:00.000Z/2992-01-10T00:00:00.000Z"],"remoteAddress":"10.0.2.2","success":"false","type":"timeseries","version":"0.9.1.1"}]
2017-03-29T03:37:57,113 ERROR [qtp448782465-23[timeseries_foodmart_1910fc54-91e6-4fd1-96eb-2147775f5027]] io.druid.server.QueryResource - Exception handling request: {class=io.druid.server.QueryResource, exceptionType=class java.lang.NullPointerException, exceptionMessage=null, exception=java.lang.NullPointerException, query=TimeseriesQuery{dataSource='foodmart'
, querySegmentSpec=LegacySegmentSpec{intervals=[1900-01-09T00:00:00.000Z/2992-01-10T00:00:00.000Z]}, descending=false, dimFilter=(io.druid.query.extraction.TimeFormatExtractionFn@ea7(__time) = 1997 && (io.druid.query.extraction.TimeFormatExtractionFn@953(__time) = 4 || io.druid.query.extraction.TimeFormatExtractionFn@953(__time) = 6)), granularity='AllGranularit
y', aggregatorSpecs=[CountAggregatorFactory{name='C'}], postAggregatorSpecs=[], context={queryId=1910fc54-91e6-4fd1-96eb-2147775f5027, skipEmptyBuckets=true, timeout=300000}}, peer=10.0.2.2}
java.lang.NullPointerException
at io.druid.query.extraction.TimeFormatExtractionFn.getCacheKey(TimeFormatExtractionFn.java:81) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.query.filter.SelectorDimFilter.getCacheKey(SelectorDimFilter.java:63) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.query.filter.DimFilterCacheHelper.computeCacheKey(DimFilterCacheHelper.java:53) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.query.filter.AndDimFilter.getCacheKey(AndDimFilter.java:59) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.query.timeseries.TimeseriesQueryQueryToolChest$4.computeCacheKey(TimeseriesQueryQueryToolChest.java:137) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.query.timeseries.TimeseriesQueryQueryToolChest$4.computeCacheKey(TimeseriesQueryQueryToolChest.java:130) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.client.CachingClusteredClient.run(CachingClusteredClient.java:238) ~[druid-server-0.9.1.1.jar:0.9.1.1]
at io.druid.query.RetryQueryRunner.run(RetryQueryRunner.java:65) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.query.timeseries.TimeseriesQueryQueryToolChest$5.run(TimeseriesQueryQueryToolChest.java:229) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.query.IntervalChunkingQueryRunner.run(IntervalChunkingQueryRunner.java:69) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.query.UnionQueryRunner.run(UnionQueryRunner.java:67) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.query.ResultMergeQueryRunner.doRun(ResultMergeQueryRunner.java:43) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.query.BySegmentSkippingQueryRunner.run(BySegmentSkippingQueryRunner.java:46) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.query.FinalizeResultsQueryRunner.run(FinalizeResultsQueryRunner.java:103) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.query.CPUTimeMetricQueryRunner.run(CPUTimeMetricQueryRunner.java:72) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.query.FluentQueryRunnerBuilder$FluentQueryRunner.run(FluentQueryRunnerBuilder.java:58) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.query.BaseQuery.run(BaseQuery.java:151) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.query.BaseQuery.run(BaseQuery.java:146) ~[druid-processing-0.9.1.1.jar:0.9.1.1]
at io.druid.server.QueryResource.doPost(QueryResource.java:216) [druid-server-0.9.1.1.jar:0.9.1.1]
at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source) ~[?:?]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_72]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_72]
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) [jersey-server-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) [jersey-servlet-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) [jersey-servlet-1.19.jar:1.19]
at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) [jersey-servlet-1.19.jar:1.19]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) [javax.servlet-api-3.1.0.jar:3.1.0]
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:278) [guice-servlet-4.0-beta.jar:?]
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:268) [guice-servlet-4.0-beta.jar:?]
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180) [guice-servlet-4.0-beta.jar:?]
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93) [guice-servlet-4.0-beta.jar:?]
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) [guice-servlet-4.0-beta.jar:?]
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120) [guice-servlet-4.0-beta.jar:?]
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:132) [guice-servlet-4.0-beta.jar:?]
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:129) [guice-servlet-4.0-beta.jar:?]
at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:206) [guice-servlet-4.0-beta.jar:?]
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:129) [guice-servlet-4.0-beta.jar:?]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83) [jetty-servlets-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364) [jetty-servlets-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1125) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515) [jetty-servlet-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1059) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.Server.handle(Server.java:497) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:248) [jetty-server-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540) [jetty-io-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540) [jetty-util-9.2.5.v20141112.jar:9.2.5.v20141112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_72]
We can make 0.9.2 the "official" version of Druid as soon as calcite-test-dataset works reliably. Since we speak REST to Druid there is no java library to update. But the version number is mentioned once or twice in the doc, IIRC.
slim bouguerra, let's check this one in now that we have moved to 0.9.2 and CALCITE-1725 is in. Can you rebase the PR and verify that tests pass? Thanks
Jesus Camacho Rodriguez i have created a new PR https://github.com/apache/calcite/pull/426.
slim bouguerra, I have just reviewed your PR and left a few comments. Overall it looks good! Please, address the comments and I will check it in.
Resolved in release 1.13.0 (2017-06-26).
https://github.com/apache/calcite/pull/406