Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8388

Misleading error message when rejecting incoming RPCs b/c of memory pressure

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Bug
    • Affects Version/s: Impala 2.12.0, Impala 3.1.0, Impala 3.2.0, Impala 3.3.0
    • Fix Version/s: None
    • Component/s: Backend
    • Epic Color:
      ghx-label-7

      Description

      When running out of memory we reject incoming RPCs (expected). However, our error message assumes that the queue is full and prints it as INT_MAX (the queue size):

      void ImpalaServicePool::RejectTooBusy(kudu::rpc::InboundCall* c) {
        string err_msg =
            Substitute("$0 request on $1 from $2 dropped due to backpressure. "
                       "The service queue contains $3 items out of a maximum of $4; "
                       "memory consumption is $5.",
                       c->remote_method().method_name(),
                       service_->service_name(),
                       c->remote_address().ToString(),
                       service_queue_.estimated_queue_length(),
                       service_queue_.max_size(),             // <-- HERE
                       PrettyPrinter::Print(service_mem_tracker_->consumption(), TUnit::BYTES));
      

      The error currently looks like this:

      I0404 11:35:43.276937 54321 impala-service-pool.cc:126] EndDataStream request on impala.DataStreamService from 1.2.3.4:56789 dropped due to backpressure. The service queue is full; it has 2147483647 items. Contents of service queue:
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              lv Lars Volker
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: