Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Fixed
-
Impala 2.8.0, Impala 2.9.0
-
ghx-label-5
Description
Log file crashing frame
# # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0x000000357f88980b, pid=564763, tid=0x00007f7b0386c700 # # JRE version: Java(TM) SE Runtime Environment (8.0_162-b12) (build 1.8.0_162-b12) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.162-b12 mixed mode linux-amd64 compressed oops) # Problematic frame: # C [libc.so.6+0x8980b] memcpy+0x15b
Crashing stack, extracted from core dump
#10 0x00007f4d8eaadbe7 in os::print_location(outputStream*, long, bool) () from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so #11 0x00007f4d8eabcaf5 in os::print_register_info(outputStream*, void*) () from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so #12 0x00007f4d8ec595a3 in VMError::report(outputStream*) () from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so #13 0x00007f4d8ec5ab2a in VMError::report_and_die() () from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so #14 0x00007f4d8eabd22f in JVM_handle_linux_signal () from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so #15 0x00007f4d8eab3253 in signalHandler(int, siginfo*, void*) () from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so #16 <signal handler called> #17 0x0000003b4d089750 in memcpy () from /lib64/libc.so.6 #18 0x0000000000845578 in impala::Expr::GetConstVal (this=0x7f430831f400, state=0x7f4cdc91b750, context=0xe331540, const_val=Unhandled dwarf expression opcode 0xf3 ) at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/exprs/expr.cc:577 #19 0x00000000008909b9 in impala::ScalarFnCall::Open (this=0x7f430831e600, state=0x7f4cdc91b750, ctx=0xe331540, scope=impala_udf::FunctionContext::FRAGMENT_LOCAL) at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/exprs/scalar-fn-call.cc:189 #20 0x000000000084af8c in impala::ExprContext::Open (this=Unhandled dwarf expression opcode 0xf3 ) at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/exprs/expr-context.cc:70 #21 0x0000000000ab2a3f in Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow (env=0xcca31f8, caller_class=Unhandled dwarf expression opcode 0xf3 ) at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/service/fe-support.cc:142 #22 0x00007f4d7b284dad in ?? () #23 0x000000059cabbe18 in ?? () #24 0x000000059cabfcd8 in ?? () #25 0xb395702563a2136b in ?? () #26 0x00000000806394b0 in ?? () #27 0xb395701200000002 in ?? () #28 0x000000059cab8090 in ?? () #29 0x00000000802f3c08 in ?? () #30 0x000000059beef118 in ?? () #31 0x00007f4cdc91bf70 in ?? () #32 0x00007f4d7b28033c in ?? () #33 0x000000059cab8438 in ?? () #34 0x000000008d567eb0 in ?? () #35 0x000000059cab8588 in ?? () #36 0x000000059cab8308 in ?? () #37 0x000000059cab85a0 in ?? () #38 0x000000059cab85d0 in ?? () #39 0x0000001811aad009 in ?? () #40 0x00000008ffffffff in ?? ()
Missing frames are from the JVM and are below (extracted from hs_err_pid file)
J 12167 org.apache.impala.service.FeSupport.NativeEvalExprsWithoutRow([B[B)[B (0 bytes) @ 0x00007f7bad2e1cf3 [0x00007f7bad2e1c80+0x73] J 12158 C1 org.apache.impala.service.FeSupport.EvalExprWithoutRow(Lorg/apache/impala/analysis/Expr;Lorg/apache/impala/thrift/TQueryCtx;)Lorg/apache/impala/thrift/TColumnValue; (170 bytes) @ 0x00007f7bad307bf4 [0x00007f7bad305be0+0x2014] J 12206 C1 org.apache.impala.service.FeSupport.EvalPredicate(Lorg/apache/impala/analysis/Expr;Lorg/apache/impala/thrift/TQueryCtx;)Z (60 bytes) @ 0x00007f7bad32daac [0x00007f7bad32d180+0x92c] J 12207 C1 org.apache.impala.analysis.Analyzer.isTrueWithNullSlots(Lorg/apache/impala/analysis/Expr;)Z (137 bytes) @ 0x00007f7bad331c54 [0x00007f7bad32fe40+0x1e14] j org.apache.impala.planner.HdfsScanNode.computeDictionaryFilterConjuncts(Lorg/apache/impala/analysis/Analyzer;)V+135 j org.apache.impala.planner.HdfsScanNode.init(Lorg/apache/impala/analysis/Analyzer;)V+22 j org.apache.impala.planner.SingleNodePlanner.createHdfsScanPlan(Lorg/apache/impala/analysis/TableRef;ZLjava/util/List;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+306 j org.apache.impala.planner.SingleNodePlanner.createScanNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+143 j org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+14 j org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41 j org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203 j org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31 j org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208 j org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107 j org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41 j org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203 j org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31 j org.apache.impala.planner.SingleNodePlanner.createUnionPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/UnionStmt;Ljava/util/List;Lorg/apache/impala/planner/PlanNode;)Lorg/apache/impala/planner/UnionNode;+141 j org.apache.impala.planner.SingleNodePlanner.createUnionPlan(Lorg/apache/impala/analysis/UnionStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+164 j org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+144 j org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208 j org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107 j org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41 j org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203 j org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31 j org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208 j org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107 j org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41 j org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203 j org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31 j org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208 j org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107 j org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41 j org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203 j org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31 j org.apache.impala.planner.SingleNodePlanner.createSingleNodePlan()Lorg/apache/impala/planner/PlanNode;+104 j org.apache.impala.planner.Planner.createPlan()Ljava/util/ArrayList;+25 j org.apache.impala.service.Frontend.createExecRequest(Lorg/apache/impala/planner/Planner;Ljava/lang/StringBuilder;)Lorg/apache/impala/thrift/TQueryExecRequest;+111 J 12874 C1 org.apache.impala.service.Frontend.createExecRequest(Lorg/apache/impala/thrift/TQueryCtx;Ljava/lang/StringBuilder;)Lorg/apache/impala/thrift/TExecRequest; (956 bytes) @ 0x00007f7bad587174 [0x00007f7bad583780+0x39f4] J 13160 C1 org.apache.impala.service.JniFrontend.createExecRequest([B)[B (100 bytes) @ 0x00007f7bad687d7c [0x00007f7bad687760+0x61c]
So the root cause seems to be in the memcpy() in the following piece of code in expr.cc
case TYPE_VARCHAR: { StringVal* sv = reinterpret_cast<StringVal*>(*const_val); *sv = GetStringVal(context, NULL); if (sv->len > 0) { // Make sure the memory is owned by 'context'. uint8_t* ptr_copy = context->pool_->TryAllocate(sv->len); if (ptr_copy == NULL) { return context->pool_->mem_tracker()->MemLimitExceeded( state, "Could not allocate constant string value", sv->len); } memcpy(ptr_copy, sv->ptr, sv->len); <--- CRASH since sv->ptr = NULL an sv->len > 0 sv->ptr = ptr_copy; } break; }
Few observations:
- The query crashes the coordinator during the query compilation/analysis (as evident from the JVM stack trace)
- The root cause seems to be due to a malformed StringVal (ptr = NULL and len >0) returned by GetStringVal and it is unclear at this point which specific function/piece of code is generating that.
- In this particular case, I figured that the ScalarFn in the crashing stack that is calling GetConstVal is concat() and removing it doesn't crash the coordinator.
- Unable to reproduce it locally on my dev box
- The problematic piece of code memcpy'ing the NULL ptr is introduced by
IMPALA-4302and removed byIMPALA-4192. Hence only 2.9.0 and 2.10.0 are the affected versions
Next Steps:
- Avoid the crash by having a stricter is_null check on the output StringVal
- Figure out which possible builtins can generate such StringVals.
Attachments
Issue Links
- is broken by
-
IMPALA-4302 In-predicate filters do not scale as expected with # of elements constant IN-list
- Resolved