Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6873

Crash in Expr::GetConstVal() due to NULL dereference

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • Impala 2.8.0, Impala 2.9.0
    • Impala 2.10.0
    • Backend
    • ghx-label-5

    Description

      Log file crashing frame

      #
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x000000357f88980b, pid=564763, tid=0x00007f7b0386c700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_162-b12) (build 1.8.0_162-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.162-b12 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libc.so.6+0x8980b]  memcpy+0x15b

      Crashing stack, extracted from core dump

      #10 0x00007f4d8eaadbe7 in os::print_location(outputStream*, long, bool) () from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so

#11 0x00007f4d8eabcaf5 in os::print_register_info(outputStream*, void*) () from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so

#12 0x00007f4d8ec595a3 in VMError::report(outputStream*) () from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so

#13 0x00007f4d8ec5ab2a in VMError::report_and_die() () from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so

#14 0x00007f4d8eabd22f in JVM_handle_linux_signal () from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so

#15 0x00007f4d8eab3253 in signalHandler(int, siginfo*, void*) () from /root/usr/java/latest/jre/lib/amd64/server/libjvm.so

#16 <signal handler called>

#17 0x0000003b4d089750 in memcpy () from /lib64/libc.so.6

#18 0x0000000000845578 in impala::Expr::GetConstVal (this=0x7f430831f400, state=0x7f4cdc91b750, context=0xe331540, const_val=Unhandled dwarf expression opcode 0xf3

) at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/exprs/expr.cc:577

#19 0x00000000008909b9 in impala::ScalarFnCall::Open (this=0x7f430831e600, state=0x7f4cdc91b750, ctx=0xe331540, scope=impala_udf::FunctionContext::FRAGMENT_LOCAL)

    at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/exprs/scalar-fn-call.cc:189

#20 0x000000000084af8c in impala::ExprContext::Open (this=Unhandled dwarf expression opcode 0xf3

) at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/exprs/expr-context.cc:70

#21 0x0000000000ab2a3f in Java_org_apache_impala_service_FeSupport_NativeEvalExprsWithoutRow (env=0xcca31f8, caller_class=Unhandled dwarf expression opcode 0xf3

) at /usr/src/debug/impala-2.9.0-cdh5.12.2/be/src/service/fe-support.cc:142

#22 0x00007f4d7b284dad in ?? ()

#23 0x000000059cabbe18 in ?? ()

#24 0x000000059cabfcd8 in ?? ()

#25 0xb395702563a2136b in ?? ()

#26 0x00000000806394b0 in ?? ()

#27 0xb395701200000002 in ?? ()

#28 0x000000059cab8090 in ?? ()

#29 0x00000000802f3c08 in ?? ()

#30 0x000000059beef118 in ?? ()

#31 0x00007f4cdc91bf70 in ?? ()

#32 0x00007f4d7b28033c in ?? ()

#33 0x000000059cab8438 in ?? ()

#34 0x000000008d567eb0 in ?? ()

#35 0x000000059cab8588 in ?? ()

#36 0x000000059cab8308 in ?? ()

#37 0x000000059cab85a0 in ?? ()

#38 0x000000059cab85d0 in ?? ()

#39 0x0000001811aad009 in ?? ()

#40 0x00000008ffffffff in ?? ()

       

      Missing frames are from the JVM and are below (extracted from hs_err_pid file)

      J 12167  org.apache.impala.service.FeSupport.NativeEvalExprsWithoutRow([B[B)[B (0 bytes) @ 0x00007f7bad2e1cf3 [0x00007f7bad2e1c80+0x73]
J 12158 C1 org.apache.impala.service.FeSupport.EvalExprWithoutRow(Lorg/apache/impala/analysis/Expr;Lorg/apache/impala/thrift/TQueryCtx;)Lorg/apache/impala/thrift/TColumnValue; (170 bytes) @ 0x00007f7bad307bf4 [0x00007f7bad305be0+0x2014]
J 12206 C1 org.apache.impala.service.FeSupport.EvalPredicate(Lorg/apache/impala/analysis/Expr;Lorg/apache/impala/thrift/TQueryCtx;)Z (60 bytes) @ 0x00007f7bad32daac [0x00007f7bad32d180+0x92c]
J 12207 C1 org.apache.impala.analysis.Analyzer.isTrueWithNullSlots(Lorg/apache/impala/analysis/Expr;)Z (137 bytes) @ 0x00007f7bad331c54 [0x00007f7bad32fe40+0x1e14]
j  org.apache.impala.planner.HdfsScanNode.computeDictionaryFilterConjuncts(Lorg/apache/impala/analysis/Analyzer;)V+135
j  org.apache.impala.planner.HdfsScanNode.init(Lorg/apache/impala/analysis/Analyzer;)V+22
j  org.apache.impala.planner.SingleNodePlanner.createHdfsScanPlan(Lorg/apache/impala/analysis/TableRef;ZLjava/util/List;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+306
j  org.apache.impala.planner.SingleNodePlanner.createScanNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+143
j  org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+14
j  org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
j  org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
j  org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
j  org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208
j  org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107
j  org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
j  org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
j  org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
j  org.apache.impala.planner.SingleNodePlanner.createUnionPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/UnionStmt;Ljava/util/List;Lorg/apache/impala/planner/PlanNode;)Lorg/apache/impala/planner/UnionNode;+141
j  org.apache.impala.planner.SingleNodePlanner.createUnionPlan(Lorg/apache/impala/analysis/UnionStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+164
j  org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+144
j  org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208
j  org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107
j  org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
j  org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
j  org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
j  org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208
j  org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107
j  org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
j  org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
j  org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
j  org.apache.impala.planner.SingleNodePlanner.createInlineViewPlan(Lorg/apache/impala/analysis/Analyzer;Lorg/apache/impala/analysis/InlineViewRef;)Lorg/apache/impala/planner/PlanNode;+208
j  org.apache.impala.planner.SingleNodePlanner.createTableRefNode(Lorg/apache/impala/analysis/TableRef;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+107
j  org.apache.impala.planner.SingleNodePlanner.createTableRefsPlan(Ljava/util/List;Ljava/util/List;ZLorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+41
j  org.apache.impala.planner.SingleNodePlanner.createSelectPlan(Lorg/apache/impala/analysis/SelectStmt;Lorg/apache/impala/analysis/Analyzer;)Lorg/apache/impala/planner/PlanNode;+203
j  org.apache.impala.planner.SingleNodePlanner.createQueryPlan(Lorg/apache/impala/analysis/QueryStmt;Lorg/apache/impala/analysis/Analyzer;Z)Lorg/apache/impala/planner/PlanNode;+31
j  org.apache.impala.planner.SingleNodePlanner.createSingleNodePlan()Lorg/apache/impala/planner/PlanNode;+104
j  org.apache.impala.planner.Planner.createPlan()Ljava/util/ArrayList;+25
j  org.apache.impala.service.Frontend.createExecRequest(Lorg/apache/impala/planner/Planner;Ljava/lang/StringBuilder;)Lorg/apache/impala/thrift/TQueryExecRequest;+111
J 12874 C1 org.apache.impala.service.Frontend.createExecRequest(Lorg/apache/impala/thrift/TQueryCtx;Ljava/lang/StringBuilder;)Lorg/apache/impala/thrift/TExecRequest; (956 bytes) @ 0x00007f7bad587174 [0x00007f7bad583780+0x39f4]
J 13160 C1 org.apache.impala.service.JniFrontend.createExecRequest([B)[B (100 bytes) @ 0x00007f7bad687d7c [0x00007f7bad687760+0x61c]

      So the root cause seems to be in the memcpy() in the following piece of code in expr.cc

      case TYPE_VARCHAR: {
      StringVal* sv = reinterpret_cast<StringVal*>(*const_val);
      *sv = GetStringVal(context, NULL);
      if (sv->len > 0) {
        // Make sure the memory is owned by 'context'.
        uint8_t* ptr_copy = context->pool_->TryAllocate(sv->len);
        if (ptr_copy == NULL) {
          return context->pool_->mem_tracker()->MemLimitExceeded(
              state, "Could not allocate constant string value", sv->len);
        }
        memcpy(ptr_copy, sv->ptr, sv->len);   <--- CRASH since sv->ptr = NULL an sv->len > 0
        sv->ptr = ptr_copy;
      }
      break;
    }

      Few observations:

      • The query crashes the coordinator during the query compilation/analysis (as evident from the JVM stack trace)
      • The root cause seems to be due to a malformed StringVal (ptr = NULL and len >0) returned by GetStringVal and it is unclear at this point which specific function/piece of code is generating that.
      • In this particular case, I figured that the ScalarFn in the crashing stack that is calling GetConstVal is concat() and removing it doesn't crash the coordinator.
      • Unable to reproduce it locally on my dev box
      • The problematic piece of code memcpy'ing the NULL ptr is introduced by IMPALA-4302 and removed by IMPALA-4192. Hence only 2.9.0 and 2.10.0 are the affected versions

      Next Steps:

      • Avoid the crash by having a stricter is_null check on the output StringVal
      • Figure out which possible builtins can generate such StringVals.

      Attachments

        Issue Links

          is broken by

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-4302 In-predicate filters do not scale as expected with # of elements constant IN-list

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              bharathv Bharath Vissapragada
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: