[KUDU-3288] tserver segfault when processing DeleteTablet

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.14.0
• Fix Version/s: 1.15.0
• Component/s: tserver
• Assignee: Unassigned
• Reporter: mintao
• Created: 27/May/21 08:06
• Updated: 27/May/21 08:12
• Resolved: 27/May/21 08:12

Description

In the core dump, the stack:

#0  0x000000000251e403 in kudu::MaintenanceManager::LaunchOp(kudu::MaintenanceOp*) () at /opt/kudu/kudu/src/kudu/util/maintenance_manager.cc:551
#1  0x000000000257c98e in operator() (this=0x7f4425076af0) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:260
#2  kudu::ThreadPool::DispatchThread() () at /opt/kudu/kudu/src/kudu/util/threadpool.cc:662
#3  0x0000000002573e25 in operator() (this=0x6f86fe8) at /opt/rh/devtoolset-8/root/usr/include/c++/8/bits/std_function.h:260
#4  kudu::Thread::SuperviseThread(void*) () at /opt/kudu/kudu/src/kudu/util/thread.cc:674
#5  0x00007f442c9bfe25 in start_thread () from /lib64/libpthread.so.0
#6  0x00007f442ac95bad in clone () from /lib64/libc.so.6

The local variables :

thread_id = 164113
op_instance = {thread_id = 164113,
  name = "CompactRowSetsOp(2c61e21e2e0b4caba1736b5c248dd65e)\000\000\000\000\000\000\350\270\344\002", '\000' <repeats 12 times>, "W\000\000\000\345\005\000\000\250ǻ>\001\000\000\000\260ߣi\000\000\000\000\000\030\323\347", '\000' <repeats 13 times>, "P\344\033\313\033\063\\\000\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000\064-><.h 8f6Do!^#=12=?(<I}.@\177:2r1O\263\000\000\000\000\000\000\000\263", '\000' <repeats 11 times>, "onse\001\377\a\000\000\000\000\000\000\000\000"..., duration = {static kUninitialized = -9223372036854775808,
    nano_delta_ = -9223372036854775808}, start_mono_time = {static kNanosecondsPerSecond = 1000000000, static kNanosecondsPerMillisecond = 1000000,
    static kNanosecondsPerMicrosecond = 1000, static kMicrosecondsPerSecond = 1000000, nanos_ = 32139819439241529}}
scoped_cleanup_L582 = <optimized out>
trace = <optimized out>
sw = <optimized out>

In the Tablet server's log, saw this:

I0526 09:47:39.229526 86465 tablet_replica.cc:291] T 2c61e21e2e0b4caba1736b5c248dd65e P c12ad54315b24a61b8c47ccd7a3ddf7e: stopping tablet replica
I0526 09:47:39.230662 86464 ts_tablet_manager.cc:1552] T 02e056b7c982476db5bd5249f7806cbd P c12ad54315b24a61b8c47ccd7a3ddf7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I0526 09:47:39.234947 164344 maintenance_manager.cc:373] P c12ad54315b24a61b8c47ccd7a3ddf7e: Scheduling CompactRowSetsOp(2c61e21e2e0b4caba1736b5c248dd65e): perf score=0.012862
I0526 09:47:39.234983 86465 raft_consensus.cc:2226] T 2c61e21e2e0b4caba1736b5c248dd65e P c12ad54315b24a61b8c47ccd7a3ddf7e [term 1 FOLLOWER]: Raft consensus shutting down.
I0526 09:47:39.235006 86465 raft_consensus.cc:2255] T 2c61e21e2e0b4caba1736b5c248dd65e P c12ad54315b24a61b8c47ccd7a3ddf7e [term 1 FOLLOWER]: Raft consensus is shut down!

Tablet server tried to perform RowSet Compacting on a Deleting tablet.