Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6873

Crash in Expr::GetConstVal() due to NULL dereference

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0, Impala 2.9.0
    • Fix Version/s: Impala 2.10.0
    • Component/s: Backend
    • Labels:
    • Epic Color:
      ghx-label-5

      Description

      Log file crashing frame

      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      #  SIGSEGV (0xb) at pc=0x000000357f88980b, pid=564763, tid=0x00007f7b0386c700
      #
      # JRE version: Java(TM) SE Runtime Environment (8.0_162-b12) (build 1.8.0_162-b12)
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.162-b12 mixed mode linux-amd64 compressed oops)
      # Problematic frame:
      # C  [libc.so.6+0x8980b]  memcpy+0x15b
      

      Crashing stack, extracted from core dump

      #10 0x00007f4d8eaadbe7 in os::print_location(outputStream*, long, bool) () from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so
      
      #11 0x00007f4d8eabcaf5 in os::print_register_info(outputStream*, void*) () from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so
      
      #12 0x00007f4d8ec595a3 in VMError::report(outputStream*) () from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so
      
      #13 0x00007f4d8ec5ab2a in VMError::report_and_die() () from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so
      
      #14 0x00007f4d8eabd22f in JVM_handle_linux_signal () from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so
      
      #15 0x00007f4d8eab3253 in signalHandler(int, siginfo*, void*) () from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so
      
      #16 <signal handler called>
      
      #17 0x0000003b4d089750 in memcpy () from /lib64/libc.so.6
      
      #18 0x0000000000845578 in impala::Expr::GetConstVal (this=0x7f430831f400, state=0x7f4cdc91b750, context=0xe331540, const_val=Unhandled dwarf expression opcode 0xf3
      
      ) at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/exprs/expr.cc:577
      
      #19 0x00000000008909b9 in impala::ScalarFnCall::Open (this=0x7f430831e600, state=0x7f4cdc91b750, ctx=0xe331540, scope=impala_udf::FunctionContext::FRAGMENT_LOCAL)
      
          at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/exprs/scalar-fn-call.cc:189
      
      #20 0x000000000084af8c in impala::ExprContext::Open (this=Unhandled dwarf expression opcode 0xf3
      
      ) at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/exprs/expr-context.cc:70
      
      #21 0x0000000000ab2a3f in Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow (env=0xcca31f8, caller_class=Unhandled dwarf expression opcode 0xf3
      
      ) at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/service/fe-support.cc:142
      
      #22 0x00007f4d7b284dad in ?? ()
      
      #23 0x000000059cabbe18 in ?? ()
      
      #24 0x000000059cabfcd8 in ?? ()
      
      #25 0xb395702563a2136b in ?? ()
      
      #26 0x00000000806394b0 in ?? ()
      
      #27 0xb395701200000002 in ?? ()
      
      #28 0x000000059cab8090 in ?? ()
      
      #29 0x00000000802f3c08 in ?? ()
      
      #30 0x000000059beef118 in ?? ()
      
      #31 0x00007f4cdc91bf70 in ?? ()
      
      #32 0x00007f4d7b28033c in ?? ()
      
      #33 0x000000059cab8438 in ?? ()
      
      #34 0x000000008d567eb0 in ?? ()
      
      #35 0x000000059cab8588 in ?? ()
      
      #36 0x000000059cab8308 in ?? ()
      
      #37 0x000000059cab85a0 in ?? ()
      
      #38 0x000000059cab85d0 in ?? ()
      
      #39 0x0000001811aad009 in ?? ()
      
      #40 0x00000008ffffffff in ?? ()
      
      

       

      Missing frames are from the JVM and are below (extracted from hs_err_pid file)

      J 12167  org.apache.impala.service.FeSupport.NativeEvalExprsWithoutRow([B[B)[B (0 bytes) @ 0x00007f7bad2e1cf3 [0x00007f7bad2e1c80+0x73]
      J 12158 C1 org.apache.impala.service.FeSupport.EvalExprWithoutRow(Lorg/apache/impala/analysis/Expr;Lorg/apache/impala/thrift/TQueryCtx;)Lorg/apache/impala/thrift/TColumnValue; (170 bytes) @ 0x00007f7bad307bf4 [0x00007f7bad305be0+0x2014]
      J 12206 C1 org.apache.impala.service.FeSupport.EvalPredicate(Lorg/apache/impala/analysis/Expr;Lorg/apache/impala/thrift/TQueryCtx;)Z (60 bytes) @ 0x00007f7bad32daac [0x00007f7bad32d180+0x92c]
      J 12207 C1 org.apache.impala.analysis.Analyzer.isTrueWithNullSlots(Lorg/apache/impala/analysis/Expr;)Z (137 bytes) @ 0x00007f7bad331c54 [0x00007f7bad32fe40+0x1e14]
      j  org.apache.impala.planner.HdfsScanNode.computeDictionaryFilterConjuncts(Lorg/apache/impala/analysis/Analyzer;)V+135
      j  org.apache.impala.planner.HdfsScanNode.init(Lorg/apache/impala/analysis/Analyzer;)V+22
      j  org.apache.impala.planner.SingleNodePlanner.createHdfsScanPlan(Lorg/apache/impala/analysis/TableRef;ZLjava/util/List;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+306
      j  org.apache.impala.planner.SingleNodePlanner.createScanNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+143
      j  org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+14
      j  org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
      j  org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
      j  org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
      j  org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208
      j  org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107
      j  org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
      j  org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
      j  org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
      j  org.apache.impala.planner.SingleNodePlanner.createUnionPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/UnionStmt;Ljava/util/List;Lorg/apache/impala/planner/PlanNode;)Lorg/apache/impala/planner/UnionNode;+141
      j  org.apache.impala.planner.SingleNodePlanner.createUnionPlan(Lorg/apache/impala/analysis/UnionStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+164
      j  org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+144
      j  org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208
      j  org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107
      j  org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
      j  org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
      j  org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
      j  org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208
      j  org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107
      j  org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
      j  org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
      j  org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
      j  org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208
      j  org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107
      j  org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
      j  org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
      j  org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
      j  org.apache.impala.planner.SingleNodePlanner.createSingleNodePlan()Lorg/apache/impala/planner/PlanNode;+104
      j  org.apache.impala.planner.Planner.createPlan()Ljava/util/ArrayList;+25
      j  org.apache.impala.service.Frontend.createExecRequest(Lorg/apache/impala/planner/Planner;Ljava/lang/StringBuilder;)Lorg/apache/impala/thrift/TQueryExecRequest;+111
      J 12874 C1 org.apache.impala.service.Frontend.createExecRequest(Lorg/apache/impala/thrift/TQueryCtx;Ljava/lang/StringBuilder;)Lorg/apache/impala/thrift/TExecRequest; (956 bytes) @ 0x00007f7bad587174 [0x00007f7bad583780+0x39f4]
      J 13160 C1 org.apache.impala.service.JniFrontend.createExecRequest([B)[B (100 bytes) @ 0x00007f7bad687d7c [0x00007f7bad687760+0x61c]
      

      So the root cause seems to be in the memcpy() in the following piece of code in expr.cc

      case TYPE_VARCHAR: {
            StringVal* sv = reinterpret_cast<StringVal*>(*const_val);
            *sv = GetStringVal(context, NULL);
            if (sv->len > 0) {
              // Make sure the memory is owned by 'context'.
              uint8_t* ptr_copy = context->pool_->TryAllocate(sv->len);
              if (ptr_copy == NULL) {
                return context->pool_->mem_tracker()->MemLimitExceeded(
                    state, "Could not allocate constant string value", sv->len);
              }
              memcpy(ptr_copy, sv->ptr, sv->len);   <--- CRASH since sv->ptr = NULL an sv->len > 0
              sv->ptr = ptr_copy;
            }
            break;
          }
      

      Few observations:

      • The query crashes the coordinator during the query compilation/analysis (as evident from the JVM stack trace)
      • The root cause seems to be due to a malformed StringVal (ptr = NULL and len >0) returned by GetStringVal and it is unclear at this point which specific function/piece of code is generating that.
      • In this particular case, I figured that the ScalarFn in the crashing stack that is calling GetConstVal is concat() and removing it doesn't crash the coordinator.
      • Unable to reproduce it locally on my dev box
      • The problematic piece of code memcpy'ing the NULL ptr is introduced by IMPALA-4302 and removed by IMPALA-4192. Hence only 2.9.0 and 2.10.0 are the affected versions

      Next Steps:

      • Avoid the crash by having a stricter is_null check on the output StringVal
      • Figure out which possible builtins can generate such StringVals.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                bharathv bharath v
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: