Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7069

Java UDF tests can trigger a crash in Java ClassLoader

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      I hit this crash on a GVO, but was able to reproduce it on master on my desktop.

      Repro steps:

      git checkout c1362afb9a072e49df470d9068d44cdbdf5cdec5
      ./buildall.sh -debug -noclean -notests -skiptests -ninja
      start-impala-cluster.py
      while impala-py.test tests/query_test/test_udfs.py -k 'hive or java or jar' -n4 --verbose; do date; done
      

      I generally hit the crash within a hour of looping the test.

      Stack: [0x00007fa04791f000,0x00007fa048120000],  sp=0x00007fa04811aff0,  free space=8175k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      V  [libjvm.so+0x8a8107]
      V  [libjvm.so+0x96cf5f]
      v  ~RuntimeStub::_complete_monitor_locking_Java
      J 2758 C2 java.util.concurrent.ConcurrentHashMap.putVal(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object; (362 bytes) @ 0x00007fa0c73637d4 [0x00007fa0c7362d00+0xad4]
      J 2311 C2 java.lang.ClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (122 bytes) @ 0x00007fa0c70a09a8 [0x00007fa0c70a08e0+0xc8]
      J 3953 C2 java.net.FactoryURLClassLoader.loadClass(Ljava/lang/String;Z)Ljava/lang/Class; (40 bytes) @ 0x00007fa0c71ce0f0 [0x00007fa0c71ce0a0+0x50]
      J 2987 C2 java.lang.ClassLoader.loadClass(Ljava/lang/String;)Ljava/lang/Class; (7 bytes) @ 0x00007fa0c72ddb64 [0x00007fa0c72ddb20+0x44]
      v  ~StubRoutines::call_stub
      V  [libjvm.so+0x6648eb]
      V  [libjvm.so+0x661ec4]
      V  [libjvm.so+0x662523]
      V  [libjvm.so+0x9e398d]
      V  [libjvm.so+0x9e2326]
      V  [libjvm.so+0x9e2b50]
      V  [libjvm.so+0x42c099]
      V  [libjvm.so+0x9dc786]
      V  [libjvm.so+0x6a5edf]
      V  [libjvm.so+0x6a70cb]  JVM_DefineClass+0xbb
      V  [libjvm.so+0xa31ea5]
      V  [libjvm.so+0xa37ea7]
      J 4842  sun.misc.Unsafe.defineClass(Ljava/lang/String;[BIILjava/lang/ClassLoader;Ljava/security/ProtectionDomain;)Ljava/lang/Class; (0 bytes) @ 0x00007fa0c7af120b [0x00007fa0c7af1100+0x10b]
      J 13229 C2 sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object; (5 bytes) @ 0x00007fa0c8cf2a74 [0x00007fa0c8cf2940+0x134]
      v  ~StubRoutines::call_stub
      V  [libjvm.so+0x6648eb]
      V  [libjvm.so+0x6b5949]  JVM_DoPrivileged+0x429
      J 1035  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object; (0 bytes) @ 0x00007fa0c7220c7f [0x00007fa0c7220bc0+0xbf]
      J 20421 C2 sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl; (762 bytes) @ 0x00007fa0c89bb848 [0x00007fa0c89b9da0+0x1aa8]
      J 4163 C2 sun.reflect.NativeMethodAccessorImpl.invoke(Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/Object; (104 bytes) @ 0x00007fa0c789cca8 [0x00007fa0c789c8c0+0x3e8]
      J 2379 C2 org.apache.impala.hive.executor.UdfExecutor.evaluate()V (396 bytes) @ 0x00007fa0c711c638 [0x00007fa0c711c400+0x238]
      v  ~StubRoutines::call_stub
      V  [libjvm.so+0x6648eb]
      V  [libjvm.so+0x6822d7]
      V  [libjvm.so+0x6862c9]
      C  [impalad+0x2a004fa]  JNIEnv_::CallNonvirtualVoidMethodA(_jobject*, _jclass*, _jmethodID*, jvalue const*)+0x40
      C  [impalad+0x29fe4ff]  impala::HiveUdfCall::Evaluate(impala::ScalarExprEvaluator*, impala::TupleRow const*) const+0x44b
      C  [impalad+0x29ffde9]  impala::HiveUdfCall::GetSmallIntVal(impala::ScalarExprEvaluator*, impala::TupleRow const*) const+0xbb
      C  [impalad+0x2a0948a]  impala::ScalarExprEvaluator::GetValue(impala::ScalarExpr const&, impala::TupleRow const*)+0x14c
      C  [impalad+0x2a48eb1]  impala::ScalarFnCall::EvaluateNonConstantChildren(impala::ScalarExprEvaluator*, impala::TupleRow const*) const+0x9d
      C  [impalad+0x2a4abba]  impala_udf::BooleanVal impala::ScalarFnCall::InterpretEval<impala_udf::BooleanVal>(impala::ScalarExprEvaluator*, impala::TupleRow const*) const+0x18c
      C  [impalad+0x2a4907d]  impala::ScalarFnCall::GetBooleanVal(impala::ScalarExprEvaluator*, impala::TupleRow const*) const+0x179
      C  [impalad+0x2a09c7f]  impala::ScalarExprEvaluator::GetBooleanVal(impala::TupleRow*)+0x37
      C  [impalad+0x1b70efb]  impala::ExecNode::EvalPredicate(impala::ScalarExprEvaluator*, impala::TupleRow*)+0x23
      C  [impalad+0x1b70efb]  impala::ExecNode::EvalPredicate(impala::ScalarExprEvaluator*, impala::TupleRow*)+0x23
      C  [impalad+0x1b6fdf0]  impala::ExecNode::EvalConjuncts(impala::ScalarExprEvaluator* const*, int, impala::TupleRow*)+0x42
      C  [impalad+0x1bb60e3]  impala::HdfsScanner::EvalConjuncts(impala::TupleRow*)+0x4d
      C  [impalad+0x1bb08fd]  impala::HdfsScanner::WriteCompleteTuple(impala::MemPool*, impala::FieldLocation*, impala::Tuple*, impala::TupleRow*, impala::Tuple*, unsigned char*, unsigned char*)+0x1b1
      C  [impalad+0x1bc176d]  impala::HdfsScanner::WriteAlignedTuples(impala::MemPool*, impala::TupleRow*, impala::FieldLocation*, int, int, int, int, bool)+0x1f3
      C  [impalad+0x1bf93d9]  impala::HdfsTextScanner::WriteFields(int, int, impala::MemPool*, impala::TupleRow*)+0x779
      C  [impalad+0x1bf48be]  impala::HdfsTextScanner::ProcessRange(impala::RowBatch*, int*)+0x9d8
      C  [impalad+0x1bf55e8]  impala::HdfsTextScanner::GetNextInternal(impala::RowBatch*)+0x422
      C  [impalad+0x1baf676]  impala::HdfsScanner::ProcessSplit()+0x1ea
      C  [impalad+0x1b8695c]  impala::HdfsScanNode::ProcessSplit(std::vector<impala::FilterContext, std::allocator<impala::FilterContext> > const&, impala::MemPool*, impala::io::ScanRange*, long*)+0x77a
      C  [impalad+0x1b85cfd]  impala::HdfsScanNode::ScannerThread(bool, long)+0x3cd
      

      I've only tried to hit this on Ubuntu16.04. It repros on at least two java versions: 1.8.0_171-8u171-b11-0ubuntu0.16.04.1-b11 and 1.8.0_03-Ubuntu-8u77-b03-3ubuntu3-b03 (I downgraded on my system to see if it was the new Java version).

      Attachments

        1. hs_err_pid9694.log
          161 kB
          Tim Armstrong
        2. hs_err_pid8975.log
          148 kB
          Tim Armstrong
        3. hs_err_pid29246.log
          155 kB
          Tim Armstrong
        4. hs_err_pid22764.log
          250 kB
          Tim Armstrong

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tarasbob Taras Bobrovytsky
            tarmstrong Tim Armstrong
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment