Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-6637

SIGSEGV at impala::BufferedTupleStream::FixUpStringsForRead()

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Not A Bug
    • Affects Version/s: Impala 3.0, Impala 2.12.0
    • Fix Version/s: Not Applicable
    • Component/s: Backend
    • Labels:

      Description

      When running stress test in a 140 node cluster, one of the node crashed with in BufferedTupleStream::FixUpStringsForRead()

      #
      # A fatal error has been detected by the Java Runtime Environment:
      #
      #  SIGSEGV (0xb) at pc=0x0000000001212607, pid=22971, tid=140176345585408
      #
      # JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
      # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
      # Problematic frame:
      # C  [impalad+0xe12607]  impala::BufferedTupleStream::FixUpStringsForRead(std::vector<impala::SlotDescriptor*, std::allocator<impala::SlotDescriptor*> > const&, impala::Tuple*)+0x17
      #
      # Core dump written. Default location: /var/log/impalad/core or core.22971
      #
      # If you would like to submit a bug report, please visit:
      #   http://bugreport.sun.com/bugreport/crash.jsp
      #
      
      ---------------  T H R E A D  ---------------
      
      Current thread is native thread
      
      siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0xffffffffc4cd450a
      
      Registers:
      RAX=0x00007fbe4d9f8298, RBX=0x00007fc4ca272750, RCX=0x00007fc5f914a630, RDX=0xffffffffc4cd44da
      RSP=0x00007f7d5948c3a0, RBP=0x00007f7d5948c3a0, RSI=0x0000000000000030, RDI=0x00007fc03b299b00
      R8 =0xffffffffc4cd451c, R9 =0x00007fbe4d9f82a0, R10=0x0000000000000001, R11=0x0000000000000001
      R12=0x0000000000000075, R13=0x00007fc4ca272760, R14=0x0000000000000001, R15=0x00007fc03b299b00
      RIP=0x0000000001212607, EFLAGS=0x0000000000010283, CSGSFS=0x27c4000000000033, ERR=0x0000000000000005
        TRAPNO=0x000000000000000e
      
      Top of Stack: (sp=0x00007f7d5948c3a0)
      0x00007f7d5948c3a0:   00007f7d5948c420 0000000001213071
      0x00007f7d5948c3b0:   00007f7d5948c46f 00007fc4a1753d40
      0x00007f7d5948c3c0:   00007f7d5948c470 00007fc03b299bb8
      0x00007f7d5948c3d0:   0000000000000010 00000400a1753d40
      0x00007f7d5948c3e0:   00007fc4a1753d40 00007fbd8b945ac0
      0x00007f7d5948c3f0:   00007f7d5948c540 00007fc790376000
      0x00007f7d5948c400:   00007fbd8b945ac0 00007f7d5948c540
      0x00007f7d5948c410:   00007fc92fde0000 00007fc790376358
      0x00007f7d5948c420:   00007f7d5948c4b0 0000000001042351
      0x00007f7d5948c430:   0000000000000001 00007fbd8b945ac0
      0x00007f7d5948c440:   00007f7d5948c540 00007fc92fde0000
      0x00007f7d5948c450:   00007f7d5948c460 00007fcf0b06aa47
      0x00007f7d5948c460:   00007f7d5948c490 00000035d9a03ea6
      0x00007f7d5948c470:   00007f7d5948c540 00007f7d5948c4b0
      0x00007f7d5948c480:   00007f7d5948c4b0 00007fc790376000
      0x00007f7d5948c490:   00007fbd8b945ac0 00007f7d5948c540
      0x00007f7d5948c4a0:   00007fc92fde0000 00007fc790376358
      0x00007f7d5948c4b0:   00007f7d5948c5a0 0000000001047e13
      0x00007f7d5948c4c0:   00007fc92fde0000 0000000100000000
      0x00007f7d5948c4d0:   00007f7d5948c500 00007f7d5948c4f0
      0x00007f7d5948c4e0:   00007f7d5948c5de 00007f7d5948c5f0
      0x00007f7d5948c4f0:   0000000000000000 0000000000000400
      0x00007f7d5948c500:   0000000002470482 00000000231d2d90
      0x00007f7d5948c510:   0087be8ed8dfc0af 0000000000000000
      0x00007f7d5948c520:   0000000000000000 00007f7d5948c601
      0x00007f7d5948c530:   00007fc6bbaf2ea8 0000000000000000
      0x00007f7d5948c540:   0087be8ed8dfc190 000000000000016e
      0x00007f7d5948c550:   0000000000000000 00007f7d5948c600
      0x00007f7d5948c560:   00007fb7e8b7b5c0 0000000000000000
      0x00007f7d5948c570:   00007f7d5948c580 000000001d916200
      0x00007f7d5948c580:   00007f7d5948c600 00007fb4efbf2f40
      0x00007f7d5948c590:   00007f7d5948c5f0 00007f7d5948c5de
      
      Instructions: (pc=0x0000000001212607)
      0x00000000012125e7:   2e 0f 1f 84 00 00 00 00 00 55 4c 8b 4e 08 48 8b
      0x00000000012125f7:   06 48 89 e5 4c 39 c8 74 3a 48 8b 08 48 63 71 74
      0x0000000001212607:   44 0f be 04 32 0f b6 71 78 41 85 f0 75 1c 48 63
      0x0000000001212617:   49 70 48 8b b7 b8 00 00 00 48 01 d1 48 89 31 48
      
      Stack: [0x00007f7d58a8d000,0x00007f7d5948e000],  sp=0x00007f7d5948c3a0,  free space=10236k
      Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
      C  [impalad+0xe12607]  impala::BufferedTupleStream::FixUpStringsForRead(std::vector<impala::SlotDescriptor*, std::allocator<impala::SlotDescriptor*> > const&, impala::Tuple*)+0x17
      C  [impalad+0xe13071]  impala::BufferedTupleStream::GetNext(impala::RowBatch*, bool*)+0x121
      C  [impalad+0xc42351]  impala::PartitionedHashJoinNode::NextSpilledProbeRowBatch(impala::RuntimeState*, impala::RowBatch*)+0x81
      C  [impalad+0xc47e13]  impala::PartitionedHashJoinNode::GetNext(impala::RuntimeState*, impala::RowBatch*, bool*)+0x2a3
      C  [impalad+0x7a98d2]  impala::FragmentInstanceState::ExecInternal()+0x1b2
      C  [impalad+0x7ac41f]  impala::FragmentInstanceState::Exec()+0x29f
      C  [impalad+0x79a84a]  impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)+0x24a
      C  [impalad+0x95d53f]  impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()()>, impala::ThreadDebugInfo const*, impala::Promise<long>*)+0x2ff
      C  [impalad+0x95dd3a]  boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()()>, impala::ThreadDebugInfo const*, impala::Promise<long>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long>*> > > >::run()+0x7a
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tarmstrong Tim Armstrong
                Reporter:
                kwho Michael Ho
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: