Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-7300 Clean up items from KRPC milestone 1
  3. IMPALA-5859

Better flow-control for TransmitData() with KRPC

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • Distributed Exec
    • None

    Description

      Each KRPC service by default has a fixed length queue which limits the number of incoming call requests to be processed. If the queue fills up, any incoming calls will be dropped and the RPC layer will return an error message:

        // Queue message on service queue
        boost::optional<InboundCall*> evicted;
        auto queue_status = service_queue_.Put(c, &evicted);
        if (queue_status == QUEUE_FULL) {
          RejectTooBusy(c);
          return Status::OK();
        }
      

      The current implementation of TransmitData() with KRPC will sleep for a predefined period of time before retrying the RPC. However, it may be wasteful of network bandwidth to resend the entire row batch if the remote server's service queue is still filled up.

      Instead, when the client hits the capacity limit above, it should fall back to a periodic ping RPC to check if the remote server has any space available and only when space is available in the remote service should the client send the entire row batch again.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              kwho Michael Ho
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: