[KUDU-2957] In HmsSentryConfigurations/MasterStressTest kudu-master crashes with on out-of-order notification log event

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.11.0
• Fix Version/s: NA
• Component/s: hms, master, test
• Assignee: Unassigned
• Reporter: Alexey Serbin
• Created: 25/Sep/19 00:51
• Updated: 03/Jun/20 15:48
• Resolved: 03/Jun/20 15:48

Description

The first relevant message is:

F0924 20:03:08.307613  1220 hms_notification_log_listener.cc:266] Received out-of-order notification log event (last processed event ID: 22): 22 DROP_TABLE default.table_7dad03ec77524186956c5829457d06c7
*** Check failure stack trace: ***                                              
    @     0x7f63a55ae62d  google::LogMessage::Fail() at ??:0                    
    @     0x7f63a55b064c  google::LogMessage::SendToLog() at ??:0               
    @     0x7f63a55ae189  google::LogMessage::Flush() at ??:0                   
    @     0x7f63a55ae3a1  google::LogMessage::~LogMessage() at ??:0             
    @     0x7f63a70c0c4c  kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
    @     0x7f63a70bff79  kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0

In the DEBUG build, all three kudu-master process crashes and next attempt to create a table times out:

F0924 20:08:32.662717  1116 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e221014b8c604ed0b635168473827877 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.58.126:41719,127.0.58.125:36217,127.0.58.124:33155, num_attempts: 772) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.58.126:41719,127.0.58.125:36217,127.0.58.124:33155, num_attempts: 1)

I attached the full log.