[IMPALA-10276] Release build sees SIGSEGV when updating the total time counter

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: Impala 4.0.0
• Fix Version/s: Impala 4.0.0
• Component/s: Backend
• Labels: broken-build crash flaky
• Target Version: Impala 4.0.0
• Epic Color: ghx-label-14
• Assignee: Tim Armstrong
• Reporter: Joe McDonnell
• Created: 22/Oct/20 17:27
• Updated: 20/Nov/20 01:44
• Resolved: 20/Nov/20 01:44

Description

A recent release build saw an Impalad crash with the following stack:

Crash reason:  SIGSEGV
Crash address: 0x11

#0 raise () from /lib64/libc.so.6
#1 abort () from /lib64/libc.so.6
#2 os::abort(bool) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
#3 VMError::report_and_die() () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
#4 JVM_handle_linux_signal () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
#5 signalHandler(int, siginfo*, void*) () from /usr/java/jdk1.8.0_144/jre/lib/amd64/server/libjvm.so
#6  <signal handler called>
#7 UpdateCounter (this=0x7f93adc88030) at /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/util/runtime-profile-counters.h:933
#8 impala::ScopedTimer<impala::MonotonicStopWatch>::~ScopedTimer (this=0x7f93adc88030, __in_chrg=<optimized out>) at /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/util/runtime-profile-counters.h:954
#9 impala::Coordinator::GetNext (this=this@entry=0x13ad09e00, results=results@entry=0xa7e18460, max_rows=max_rows@entry=-1, eos=eos@entry=0x7f93adc880ef, block_on_wait_time_us=block_on_wait_time_us@entry=0) at /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/runtime/coordinator.cc:864
#10 impala::ClientRequestState::FetchRowsInternal (this=this@entry=0x11568000, max_rows=max_rows@entry=-1, fetched_rows=fetched_rows@entry=0xa7e18460, block_on_wait_time_us=block_on_wait_time_us@entry=0) at /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/service/client-request-state.cc:1090
#11 impala::ClientRequestState::FetchRows (this=0x11568000, max_rows=max_rows@entry=-1, fetched_rows=fetched_rows@entry=0xa7e18460, block_on_wait_time_us=0) at /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/service/client-request-state.cc:938
#12 impala::ImpalaServer::FetchInternal (this=this@entry=0xdd35b00, query_id=..., start_over=start_over@entry=false, fetch_size=fetch_size@entry=-1, query_results=query_results@entry=0x7f93adc88498) at /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/service/impala-beeswax-server.cc:614
#13 impala::ImpalaServer::fetch (this=0xdd35b00, query_results=..., beeswax_handle=..., start_over=<optimized out>, fetch_size=-1) at /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/service/impala-beeswax-server.cc:191
#14 beeswax::BeeswaxServiceProcessor::process_fetch (this=0x102b0c60, seqid=0, iprot=<optimized out>, oprot=0xd45d0980, callContext=<optimized out>) at /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/BeeswaxService.cpp:3398
#15 beeswax::BeeswaxServiceProcessor::dispatchCall (this=0x102b0c60, iprot=0xd45d1e00, oprot=0xd45d0980, fname=..., seqid=0, callContext=0x12ea9a80) at /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/generated-sources/gen-cpp/BeeswaxService.cpp:3200
#16 apache::thrift::TDispatchProcessor::process (this=0x102b0c60, in=..., out=..., connectionContext=0x12ea9a80) at /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/Impala-Toolchain/toolchain-packages-gcc7.5.0/thrift-0.9.3-p8/include/thrift/TDispatchProcessor.h:121
#17 apache::thrift::server::TAcceptQueueServer::Task::run (this=0x11a92e80) at /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/repos/Impala/be/src/rpc/TAcceptQueueServer.cpp:84
#18 operator() (a2=<optimized out>, a1=..., p=<optimized out>, this=<optimized out>) at /data/jenkins/workspace/impala-cdpd-master-staging-exhaustive-release/Impala-Toolchain/toolchain-packages-gcc7.5.0/boost-1.61.0-p2/include/boost/bind/mem_fn_template.hpp:280

The code is updating the total time counter in the runtime profile via a ScopedTimer. 

This has been seen once

Issue Links

• is broken by: IMPALA-9382 Prototype denser runtime profile implementation
• is broken by: IMPALA-9711 Incrementally compute averaged profile