[IMPALA-6637] SIGSEGV at impala::BufferedTupleStream::FixUpStringsForRead()

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Not A Bug
• Affects Version/s: Impala 3.0, Impala 2.12.0
• Fix Version/s: Not Applicable
• Component/s: Backend
• Labels: crash
• Target Version: Impala 2.12.0
• Epic Color: ghx-label-2
• Assignee: Tim Armstrong
• Reporter: Michael Ho
• Created: 10/Mar/18 06:18
• Updated: 12/Mar/18 22:32
• Resolved: 12/Mar/18 22:29

Description

When running stress test in a 140 node cluster, one of the node crashed with in BufferedTupleStream::FixUpStringsForRead()

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000001212607, pid=22971, tid=140176345585408
#
# JRE version: Java(TM) SE Runtime Environment (7.0_67-b01) (build 1.7.0_67-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [impalad+0xe12607]  impala::BufferedTupleStream::FixUpStringsForRead(std::vector<impala::SlotDescriptor*, std::allocator<impala::SlotDescriptor*> > const&, impala::Tuple*)+0x17
#
# Core dump written. Default location: /var/log/impalad/core or core.22971
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread is native thread

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0xffffffffc4cd450a

Registers:
RAX=0x00007fbe4d9f8298, RBX=0x00007fc4ca272750, RCX=0x00007fc5f914a630, RDX=0xffffffffc4cd44da
RSP=0x00007f7d5948c3a0, RBP=0x00007f7d5948c3a0, RSI=0x0000000000000030, RDI=0x00007fc03b299b00
R8 =0xffffffffc4cd451c, R9 =0x00007fbe4d9f82a0, R10=0x0000000000000001, R11=0x0000000000000001
R12=0x0000000000000075, R13=0x00007fc4ca272760, R14=0x0000000000000001, R15=0x00007fc03b299b00
RIP=0x0000000001212607, EFLAGS=0x0000000000010283, CSGSFS=0x27c4000000000033, ERR=0x0000000000000005
  TRAPNO=0x000000000000000e

Top of Stack: (sp=0x00007f7d5948c3a0)
0x00007f7d5948c3a0:   00007f7d5948c420 0000000001213071
0x00007f7d5948c3b0:   00007f7d5948c46f 00007fc4a1753d40
0x00007f7d5948c3c0:   00007f7d5948c470 00007fc03b299bb8
0x00007f7d5948c3d0:   0000000000000010 00000400a1753d40
0x00007f7d5948c3e0:   00007fc4a1753d40 00007fbd8b945ac0
0x00007f7d5948c3f0:   00007f7d5948c540 00007fc790376000
0x00007f7d5948c400:   00007fbd8b945ac0 00007f7d5948c540
0x00007f7d5948c410:   00007fc92fde0000 00007fc790376358
0x00007f7d5948c420:   00007f7d5948c4b0 0000000001042351
0x00007f7d5948c430:   0000000000000001 00007fbd8b945ac0
0x00007f7d5948c440:   00007f7d5948c540 00007fc92fde0000
0x00007f7d5948c450:   00007f7d5948c460 00007fcf0b06aa47
0x00007f7d5948c460:   00007f7d5948c490 00000035d9a03ea6
0x00007f7d5948c470:   00007f7d5948c540 00007f7d5948c4b0
0x00007f7d5948c480:   00007f7d5948c4b0 00007fc790376000
0x00007f7d5948c490:   00007fbd8b945ac0 00007f7d5948c540
0x00007f7d5948c4a0:   00007fc92fde0000 00007fc790376358
0x00007f7d5948c4b0:   00007f7d5948c5a0 0000000001047e13
0x00007f7d5948c4c0:   00007fc92fde0000 0000000100000000
0x00007f7d5948c4d0:   00007f7d5948c500 00007f7d5948c4f0
0x00007f7d5948c4e0:   00007f7d5948c5de 00007f7d5948c5f0
0x00007f7d5948c4f0:   0000000000000000 0000000000000400
0x00007f7d5948c500:   0000000002470482 00000000231d2d90
0x00007f7d5948c510:   0087be8ed8dfc0af 0000000000000000
0x00007f7d5948c520:   0000000000000000 00007f7d5948c601
0x00007f7d5948c530:   00007fc6bbaf2ea8 0000000000000000
0x00007f7d5948c540:   0087be8ed8dfc190 000000000000016e
0x00007f7d5948c550:   0000000000000000 00007f7d5948c600
0x00007f7d5948c560:   00007fb7e8b7b5c0 0000000000000000
0x00007f7d5948c570:   00007f7d5948c580 000000001d916200
0x00007f7d5948c580:   00007f7d5948c600 00007fb4efbf2f40
0x00007f7d5948c590:   00007f7d5948c5f0 00007f7d5948c5de

Instructions: (pc=0x0000000001212607)
0x00000000012125e7:   2e 0f 1f 84 00 00 00 00 00 55 4c 8b 4e 08 48 8b
0x00000000012125f7:   06 48 89 e5 4c 39 c8 74 3a 48 8b 08 48 63 71 74
0x0000000001212607:   44 0f be 04 32 0f b6 71 78 41 85 f0 75 1c 48 63
0x0000000001212617:   49 70 48 8b b7 b8 00 00 00 48 01 d1 48 89 31 48

Stack: [0x00007f7d58a8d000,0x00007f7d5948e000],  sp=0x00007f7d5948c3a0,  free space=10236k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [impalad+0xe12607]  impala::BufferedTupleStream::FixUpStringsForRead(std::vector<impala::SlotDescriptor*, std::allocator<impala::SlotDescriptor*> > const&, impala::Tuple*)+0x17
C  [impalad+0xe13071]  impala::BufferedTupleStream::GetNext(impala::RowBatch*, bool*)+0x121
C  [impalad+0xc42351]  impala::PartitionedHashJoinNode::NextSpilledProbeRowBatch(impala::RuntimeState*, impala::RowBatch*)+0x81
C  [impalad+0xc47e13]  impala::PartitionedHashJoinNode::GetNext(impala::RuntimeState*, impala::RowBatch*, bool*)+0x2a3
C  [impalad+0x7a98d2]  impala::FragmentInstanceState::ExecInternal()+0x1b2
C  [impalad+0x7ac41f]  impala::FragmentInstanceState::Exec()+0x29f
C  [impalad+0x79a84a]  impala::QueryState::ExecFInstance(impala::FragmentInstanceState*)+0x24a
C  [impalad+0x95d53f]  impala::Thread::SuperviseThread(std::string const&, std::string const&, boost::function<void ()()>, impala::ThreadDebugInfo const*, impala::Promise<long>*)+0x2ff
C  [impalad+0x95dd3a]  boost::detail::thread_data<boost::_bi::bind_t<void, void (*)(std::string const&, std::string const&, boost::function<void ()()>, impala::ThreadDebugInfo const*, impala::Promise<long>*), boost::_bi::list5<boost::_bi::value<std::string>, boost::_bi::value<std::string>, boost::_bi::value<boost::function<void ()()> >, boost::_bi::value<impala::ThreadDebugInfo*>, boost::_bi::value<impala::Promise<long>*> > > >::run()+0x7a

Issue Links

• relates to: IMPALA-6645 Consider enabling disk spill encryption by default