[IMPALA-9572] Impalad crash when process decimal value

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 4.0.0, Impala 3.4.0, Impala 3.4.1
• Fix Version/s: Impala 4.0.0, Impala 3.4.2
• Component/s: Backend
• Labels: broken-build crash
• Epic Color: ghx-label-12
• Assignee: Csaba Ringhofer
• Reporter: Yongzhi Chen
• Created: 30/Mar/20 15:22
• Updated: 10/Mar/24 12:26
• Resolved: 14/Aug/20 11:11

Description

In impala-asf-master-exhaustive build, minidump shows impalad crashed:
Crash reason: SIGABRT
Crash address: 0x7d10000472e
Process uptime: not available

Thread 396 (crashed)
0 libc-2.17.so + 0x351f7
rax = 0x0000000000000000 rdx = 0x0000000000000006
rcx = 0xffffffffffffffff rbx = 0x0000000007246600
rsi = 0x0000000000004273 rdi = 0x000000000000472e
rbp = 0x00007f6e43ee7050 rsp = 0x00007f6e43ee6ce8
r8 = 0x0000000000000000 r9 = 0x00007f6e43ee6b60
r10 = 0x0000000000000008 r11 = 0x0000000000000206
r12 = 0x0000000007246680 r13 = 0x0000000000000044
r14 = 0x000000000724dfc4 r15 = 0x0000000007246600
rip = 0x00007f6f2b1da1f7
Found by: given as instruction pointer in context
1 libc-2.17.so + 0x368e8
rbp = 0x00007f6e43ee7050 rsp = 0x00007f6e43ee6cf0
rip = 0x00007f6f2b1db8e8
Found by: stack scanning
2 impalad!google_breakpad::ExceptionHandler::HandleSignal(int, siginfo_t*, void*) + 0x1e0
rbp = 0x00007f6e43ee7050 rsp = 0x00007f6e43ee6d78
rip = 0x0000000004ed0840
Found by: stack scanning
3 impalad!google::DumpStackTraceAndExit() + 0x24
rbp = 0x00007f6e43ee7050 rsp = 0x00007f6e43ee6e20
rip = 0x0000000004ea2554
Found by: stack scanning
4 impalad!google::LogMessage::Fail() + 0xd
rbx = 0x0000000007246600 rbp = 0x00007f6e43ee7050
rsp = 0x00007f6e43ee6ed0 rip = 0x0000000004e98fad
Found by: call frame info
5 impalad!google::LogMessage::SendToLog() + 0x2b2
rbx = 0x0000000007246600 rbp = 0x00007f6e43ee7050
rsp = 0x00007f6e43ee6ee0 rip = 0x0000000004e9a852
Found by: call frame info
6 impalad!google::LogMessage::Flush() + 0x157
rbx = 0x00007f6e43ee7090 rbp = 0x00007f6f2bd8a6a0
rsp = 0x00007f6e43ee7060 r12 = 0x00007f6e43ee707f
r13 = 0x00000000072554f8 r14 = 0x00007f6e43ee7120
r15 = 0x0000000018954c50 rip = 0x0000000004e98987
Found by: call frame info
7 impalad!google::LogMessageFatal::~LogMessageFatal() + 0xe
rbx = 0x00007f6e43ee7120 rbp = 0x00007f6e43ee7160
rsp = 0x00007f6e43ee70e0 r12 = 0x0000000000000001
r13 = 0x00000000072554f8 r14 = 0x0000000118ba0000
r15 = 0x0000000018954c50 rip = 0x0000000004e9bf4e
Found by: call frame info
8 impalad!impala::StrideWriter<long>::StrideWriter(long*, long) [mem-util.h : 40 + 0xd]
rbx = 0x0000000000000001 rbp = 0x00007f6e43ee7160
rsp = 0x00007f6e43ee7100 r12 = 0x0000000000000001
r13 = 0x00000000072554f8 r14 = 0x0000000118ba0000
r15 = 0x0000000018954c50 rip = 0x0000000002c07535
Found by: call frame info
9 impalad!impala::BaseScalarColumnReader::FillPositionsInCandidateRange(int, int, unsigned char*, int) [parquet-column-readers.cc : 1319 + 0x1f]
rbx = 0x00000000000017d4 rbp = 0x00007f6e43ee7270
rsp = 0x00007f6e43ee7170 r12 = 0x0000000000000000
r13 = 0x0000000000000039 r14 = 0x0000000118ba0000
r15 = 0x0000000018954c50 rip = 0x0000000002c008d3
Found by: call frame info
10 impalad!impala::ScalarColumnReader<impala::DecimalValue<long>, (parquet::Type::type)7, true>::MaterializeValueBatchRepeatedDefLevel(int, int, unsigned char*, int*) [parquet-column-readers.cc : 612 + 0x2b]
rbx = 0x0000000000000007 rbp = 0x00007f6e43ee73a0
rsp = 0x00007f6e43ee7280 r12 = 0x0000000000000000
r13 = 0x0000000000000039 r14 = 0x0000000118ba0000
r15 = 0x0000000018954c50 rip = 0x0000000002c731f7
Found by: call frame info
11 impalad!bool impala::ScalarColumnReader<impala::DecimalValue<long>, (parquet::Type::type)7, true>::ReadValueBatch<true>(int, int, unsigned char*, int*) [parquet-column-readers.cc : 468 + 0x26]
rbx = 0x0000000000000000 rbp = 0x00007f6e43ee7520
rsp = 0x00007f6e43ee73b0 r12 = 0x0000000000000400
r13 = 0x0000000000000039 r14 = 0x0000000118ba0000
r15 = 0x0000000018954c50 rip = 0x0000000002c54db1
Found by: call frame info
12 impalad!impala::ScalarColumnReader<impala::DecimalValue<long>, (parquet::Type::type)7, true>::ReadValueBatch(impala::MemPool*, int, int, unsigned char*, int*) [parquet-column-readers.cc : 77 + 0x1d]
rbx = 0x0000000002c1cdac rbp = 0x00007f6e43ee7560
rsp = 0x00007f6e43ee7530 r12 = 0x0000000000000400
r13 = 0x0000000000000039 r14 = 0x0000000118ba0000
r15 = 0x0000000018954c50 rip = 0x0000000002c1cde7
Found by: call frame info
13 impalad!impala::HdfsParquetScanner::AssembleRows(std::vector<impala::ParquetColumnReader*, std::allocator<impala::ParquetColumnReader*> > const&, impala::RowBatch*, bool*) [hdfs-parquet-scanner.cc : 1111 + 0x19]
rbx = 0x0000000002c1cdac rbp = 0x00007f6e43ee7880
rsp = 0x00007f6e43ee7570 r12 = 0x0000000000000400
r13 = 0x0000000000000039 r14 = 0x0000000118ba0000
r15 = 0x0000000018954c50 rip = 0x0000000002bb4145
Found by: call frame info
14 impalad!impala::HdfsParquetScanner::GetNextInternal(impala::RowBatch*) [hdfs-parquet-scanner.cc : 454 + 0x42]
rbx = 0x00007f6e43ee8080 rbp = 0x00007f6e43ee7f30
rsp = 0x00007f6e43ee7890 r12 = 0x0000000000000000
r13 = 0x0000000000000000 r14 = 0x00000000a6971fc8
r15 = 0x000000000000002c rip = 0x0000000002baefe5
Found by: call frame info
15 impalad!impala::HdfsParquetScanner::ProcessSplit() [hdfs-parquet-scanner.cc : 351 + 0x39]
rbx = 0x0000000002bad198 rbp = 0x00007f6e43ee8000

There are several test failures after that with same error:
Query aborted:Failed due to unreachable impalad(s): impala-ec2-centos74-m5-4xlarge-ondemand-1a73.vpc.cloudera.com:22001

The first related test may be:
query_test.test_spilling.TestSpillingDebugActionDimensions.test_spilling