Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1632

Assert on request size in consensus_queue.cc

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 1.0.0
    • Fix Version/s: n/a
    • Component/s: consensus
    • Labels:
      None
    • Environment:
      Development environment: Linux or MacOS X will do. Build the project in debug configuration.

      Description

      After switching tests from running in MANUAL_FLUSH mode into AUTO_FLUSH_BACKGROUND mode, the debug assert started to fire when running the all_types-itest built in debug configuration and KUDU_ALLOW_SLOW_TESTS=1. It's 100% reproducible: just run

      KUDU_ALLOW_SLOW_TESTS=1 ./bin/all_types-itest  2>/tmp/all_types.log
      

      The following stacktrace is reported by the test:

      F0920 12:26:24.726744 13286 consensus_queue.cc:401] Check failed: request->ByteSize() <= FLAGS_consensus_max_batch_size_bytes (1168286 vs. 1048576)
      *** Check failure stack trace: ***
          @     0x7f74c9f42294  google::LogMessage::SendToLog()
          @     0x7f74c9f42790  google::LogMessage::Flush()
          @     0x7f74c9f463c2  google::LogMessageFatal::~LogMessageFatal()
          @     0x7f74cf109918 kudu::consensus::PeerMessageQueue::RequestForPeer()
          @     0x7f74cf0fc302  kudu::consensus::Peer::SendNextRequest()
          @     0x7f74cf0fdfaf  kudu::consensus::Peer::DoProcessResponse()
          @     0x7f74cf1050a9  kudu::internal::RunnableAdapter<>::Run()
          @     0x7f74cf10502c  kudu::internal::InvokeHelper<>::MakeItSo()
          @     0x7f74cf104ffa  kudu::internal::Invoker<>::Run()
          @     0x7f74cf1284ce  kudu::Callback<>::Run()
          @     0x7f74cf12e7b9  boost::_mfi::cmf0<>::operator()()
          @     0x7f74cf12e720  boost::_bi::list1<>::operator()<>()
          @     0x7f74cf12e6ca  boost::_bi::bind_t<>::operator()()
          @     0x7f74cf12e430 boost::detail::function::void_function_obj_invoker0<>::invoke()
          @     0x7f74ce164b98  boost::function0<>::operator()()
          @     0x7f74cab47589  kudu::FunctionRunnable::Run()
          @     0x7f74cab45994  kudu::ThreadPool::DispatchThread()
          @     0x7f74cab49cc9  boost::_mfi::mf1<>::operator()()
          @     0x7f74cab49c27  boost::_bi::list2<>::operator()<>()
          @     0x7f74cab49baa  boost::_bi::bind_t<>::operator()()
          @     0x7f74cab49930 boost::detail::function::void_function_obj_invoker0<>::invoke()
          @     0x7f74ce164b98  boost::function0<>::operator()()
          @     0x7f74cab3ab30  kudu::Thread::SuperviseThread()
          @       0x3ae0e079d1  (unknown)
          @       0x3ae0ae88fd  (unknown)
          @              (nil)  (unknown)
      

      The patch which introduces the change from MANUAL_FLUSH to AUTO_FLUSH_BACKGROUND can be found at: https://gerrit.cloudera.org/#/c/4471/

      Also, if trying to reproduce the issue, set the consensus_max_batch_size_bytes to 1048576 (i.e. 1MiB): as a temporary workaround to avoid triggering the debug assert, the all_types-itest set it to 2MiB.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                aserbin Alexey Serbin
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: