[DRILL-6639] Exception happens while displaying operator profiles for some queries

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.14.0
• Fix Version/s: 1.15.0
• Labels: ready-to-commit
• Assignee: Vitalii Diravka
• Reporter: Anton Gozhiy
• Reviewer: Arina Ielchiieva
• Created: 26/Jul/18 16:21
• Updated: 21/Aug/18 14:12
• Resolved: 30/Jul/18 21:48

Description

Prerequisites:
1. Create a MapR-DB JSON table:

hadoop fs -mkdir /tmp/mdb_tabl
mapr dbshell
create /tmp/mdb_table/json
insert /tmp/mdb_table/json --value '{"_id":"movie0000002" , "title":"Developers on the Edge", "studio":"Command Line Studios"}'
insert /tmp/mdb_table/json --id movie0000003 --value '{"title":"The Golden Master", "studio":"All-Nighter"}'

2. Create a Hive external table:

CREATE EXTERNAL TABLE mapr_db_json_hive_tbl ( 
movie_id string, title string, studio string) 
STORED BY 'org.apache.hadoop.hive.maprdb.json.MapRDBJsonStorageHandler' 
TBLPROPERTIES("maprdb.table.name" = "/tmp/mdb_table/json","maprdb.column.id" = "movie_id");

3. Enable Hive storage plugin in Drill:

{
  "type": "hive",
  "enabled": true,
  "configProps": {
	"hive.metastore.uris": "thrift://localhost:9083",
	"fs.default.name": "maprfs:///",
	"hive.metastore.sasl.enabled": "false"
  }
}

Steps:
1. Run the following query:

select * from hive.`mapr_db_json_hive_tbl`

2. Open the query profile in the Drill UI, look at the Operator Profiles

Expected result:
Operator Profiles should be displayed

Actual result:
Exception displayed:

FreeMarker template error (DEBUG mode; use RETHROW in production!): Java method "org.apache.drill.exec.server.rest.profile.ProfileWrapper.getOperatorsOverview()" threw an exception when invoked on org.apache.drill.exec.server.rest.profile.ProfileWrapper object "org.apache.drill.exec.server.rest.profile.ProfileWrapper@36c94e5"; see cause exception in the Java stack trace. ---- FTL stack trace ("~" means nesting-related): - Failed at: ${model.getOperatorsOverview()?no_esc} [in template "rest/profile/profile.ftl" in macro "page_body" at line 338, column 11] - Reached through: @page_body [in template "rest/generic.ftl" in macro "page_html" at line 99, column 9] - Reached through: @page_html [in template "rest/profile/profile.ftl" at line 474, column 1] ---- Java stack trace (for programmers): ---- freemarker.core._TemplateModelException: [... Exception message was already printed; see it above ...] at freemarker.ext.beans._MethodUtil.newInvocationTemplateModelException(_MethodUtil.java:289) at freemarker.ext.beans._MethodUtil.newInvocationTemplateModelException(_MethodUtil.java:252) at freemarker.ext.beans.SimpleMethodModel.exec(SimpleMethodModel.java:74) at freemarker.core.MethodCall._eval(MethodCall.java:65) at freemarker.core.Expression.eval(Expression.java:81) at freemarker.core.BuiltInsForOutputFormatRelated$AbstractConverterBI.calculateResult(BuiltInsForOutputFormatRelated.java:50) at freemarker.core.MarkupOutputFormatBoundBuiltIn._eval(MarkupOutputFormatBoundBuiltIn.java:40) at freemarker.core.Expression.eval(Expression.java:81) at freemarker.core.DollarVariable.calculateInterpolatedStringOrMarkup(DollarVariable.java:96) at freemarker.core.DollarVariable.accept(DollarVariable.java:59) at freemarker.core.Environment.visit(Environment.java:362) at freemarker.core.Environment.invoke(Environment.java:714) at freemarker.core.UnifiedCall.accept(UnifiedCall.java:83) at freemarker.core.Environment.visit(Environment.java:362) at freemarker.core.Environment.invoke(Environment.java:714) at freemarker.core.UnifiedCall.accept(UnifiedCall.java:83) at freemarker.core.Environment.visit(Environment.java:326) at freemarker.core.Environment.visit(Environment.java:332) at freemarker.core.Environment.process(Environment.java:305) at freemarker.template.Template.process(Template.java:378) at org.glassfish.jersey.server.mvc.freemarker.FreemarkerViewProcessor.writeTo(FreemarkerViewProcessor.java:143) at org.glassfish.jersey.server.mvc.freemarker.FreemarkerViewProcessor.writeTo(FreemarkerViewProcessor.java:85) at org.glassfish.jersey.server.mvc.spi.ResolvedViewable.writeTo(ResolvedViewable.java:116) at org.glassfish.jersey.server.mvc.internal.ViewableMessageBodyWriter.writeTo(ViewableMessageBodyWriter.java:134) at org.glassfish.jersey.server.mvc.internal.ViewableMessageBodyWriter.writeTo(ViewableMessageBodyWriter.java:88) at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.invokeWriteTo(WriterInterceptorExecutor.java:263) at org.glassfish.jersey.message.internal.WriterInterceptorExecutor$TerminalWriterInterceptor.aroundWriteTo(WriterInterceptorExecutor.java:250) at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) at org.glassfish.jersey.server.mvc.internal.TemplateMethodInterceptor.aroundWriteTo(TemplateMethodInterceptor.java:77) at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) at org.glassfish.jersey.server.internal.JsonWithPaddingInterceptor.aroundWriteTo(JsonWithPaddingInterceptor.java:103) at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) at org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundWriteTo(MappableExceptionWrapperInterceptor.java:88) at org.glassfish.jersey.message.internal.WriterInterceptorExecutor.proceed(WriterInterceptorExecutor.java:162) at org.glassfish.jersey.message.internal.MessageBodyFactory.writeTo(MessageBodyFactory.java:1154) at org.glassfish.jersey.server.ServerRuntime$Responder.writeResponse(ServerRuntime.java:613) at org.glassfish.jersey.server.ServerRuntime$Responder.processResponse(ServerRuntime.java:375) at org.glassfish.jersey.server.ServerRuntime$Responder.process(ServerRuntime.java:365) at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:272) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:267) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:297) at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:252) at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:1023) at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:372) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:382) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:345) at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:220) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:738) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:551) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:219) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1111) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:478) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1045) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97) at org.eclipse.jetty.server.Server.handle(Server.java:462) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:279) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:232) at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:534) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:607) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:536) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.NullPointerException at org.apache.drill.exec.server.rest.profile.OperatorWrapper.getSpillCycleMetricIndex(OperatorWrapper.java:258) at org.apache.drill.exec.server.rest.profile.OperatorWrapper.addSummary(OperatorWrapper.java:161) at org.apache.drill.exec.server.rest.profile.ProfileWrapper.getOperatorsOverview(ProfileWrapper.java:291) at sun.reflect.GeneratedMethodAccessor196.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at freemarker.ext.beans.BeansWrapper.invokeMethod(BeansWrapper.java:1487) at freemarker.ext.beans.SimpleMethodModel.exec(SimpleMethodModel.java:69) ... 64 more

Issue Links

• is related to: DRILL-6643 OperatorType for Values and IteratorValidator
• is related to: DRILL-6642 Update protocol-buffers version
• relates to: DRILL-6279 Web UI should indicate when operators have spilled in-memory data to disk
• links to: GitHub Pull Request #1404