[IMPALA-8539] ASAN heap-use-after-free failure during TestGracefulShutdown

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: Impala 3.3.0
• Fix Version/s: Impala 4.0.0
• Component/s: Backend
• Epic Color: ghx-label-9
• Assignee: Bikramjeet Vig
• Reporter: David Rorke
• Created: 11/May/19 00:33
• Updated: 16/Jul/20 20:28
• Resolved: 16/Jul/20 20:28

Description

I'm seeing an ASAN heap-use-after-free failure with no stack trace in the stderr.  Here's the full ERROR log:

Log file created at: 2019/05/10 06:32:22
Running on machine: impala-ec2-centos74-r4-4xlarge-ondemand-08f1.vpc.cloudera.com
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0510 06:32:22.212716 102164 logging.cc:147] stderr will be logged to this file.
19/05/10 06:32:23 INFO util.JvmPauseMonitor: Starting JVM pause monitor
=================================================================
==102164==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000032668 at pc 0x0000017aa3dc bp 0x7ff292197ae0 sp 0x7ff292197290
READ of size 4 at 0x603000032668 thread T3
Picked up JAVA_TOOL_OPTIONS: -agentlib:jdwp=transport=dt_socket,address=30000,server=y,suspend=n

With no stack trace it's hard to say exactly what triggered it but correlating the timing of the error log entry with the Jenkins console logs it looks like we were running one of the graceful shutdown tests:

06:32:19 custom_cluster/test_restart_services.py::TestGracefulShutdown::test_sending_multiple_shutdown_signals PASSED
06:32:37 custom_cluster/test_restart_services.py::TestGracefulShutdown::test_shutdown_coordinator <- hs2/hs2_test_suite.py PASSED
06:32:37 custom_cluster/test_rpc_exception.py::TestRPCException::test_rpc_send_closed_connection[protocol: beeswax | exec_option: {'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0, 'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0} | table_format: text/none] SKIPPED

Issue Links

• is duplicated by: IMPALA-9739 TSAN data races during impalad shutdown
• is duplicated by: IMPALA-9344 TestGracefulShutdown hits ASAN use after free