Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2957

In HmsSentryConfigurations/MasterStressTest kudu-master crashes with on out-of-order notification log event

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 1.11.0
    • Fix Version/s: NA
    • Component/s: hms, master, test
    • Labels:
      None

      Description

      The first relevant message is:

      F0924 20:03:08.307613  1220 hms_notification_log_listener.cc:266] Received out-of-order notification log event (last processed event ID: 22): 22 DROP_TABLE default.table_7dad03ec77524186956c5829457d06c7
      *** Check failure stack trace: ***                                              
          @     0x7f63a55ae62d  google::LogMessage::Fail() at ??:0                    
          @     0x7f63a55b064c  google::LogMessage::SendToLog() at ??:0               
          @     0x7f63a55ae189  google::LogMessage::Flush() at ??:0                   
          @     0x7f63a55ae3a1  google::LogMessage::~LogMessage() at ??:0             
          @     0x7f63a70c0c4c  kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
          @     0x7f63a70bff79  kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
      

      In the DEBUG build, all three kudu-master process crashes and next attempt to create a table times out:

      F0924 20:08:32.662717  1116 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_e221014b8c604ed0b635168473827877 on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.0.58.126:41719,127.0.58.125:36217,127.0.58.124:33155, num_attempts: 772) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.0.58.126:41719,127.0.58.125:36217,127.0.58.124:33155, num_attempts: 1)
      

      I attached the full log.

        Attachments

        1. master-stress-test.1.txt.xz
          59 kB
          Alexey Serbin

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              aserbin Alexey Serbin
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: