Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5217

Heartbeat Fails when C++ client receives a large ResultSet

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • None
    • None
    • Client - C++
    • None

    Description

      If the listener thread is occupied for longer than 15 seconds (heartbeat timeout) while handling a message from the drillbit e.g. processing query data blocks if the query result listener's buffer is full, heartbeats fail because the same thread is responsible for sending heartbeats!

      Fix is to handle long running operations separately using boost asio.

      Attachments

        Issue Links

          Activity

            laurent Laurent Goujon added a comment -

            Are you suggesting to offset the notification to the client to a different threadpool (or ioservice)?

            laurent Laurent Goujon added a comment - Are you suggesting to offset the notification to the client to a different threadpool (or ioservice)?

            Not reception of "pong" notification, but sending the "ping" notification.

            The timer that sends the "ping" uses the same ioservice (for async_wait), and the ioservice is composed of 1 thread, which is essentially blocked. The mentioned (suggested) fix avoids the alternative, which is to create another thread that is only responsible for sending pings. Do you have suggestions?

            sudheeshkatkam Sudheesh Katkam added a comment - Not reception of "pong" notification, but sending the "ping" notification. The timer that sends the "ping" uses the same ioservice (for async_wait), and the ioservice is composed of 1 thread, which is essentially blocked. The mentioned (suggested) fix avoids the alternative, which is to create another thread that is only responsible for sending pings. Do you have suggestions?
            laurent Laurent Goujon added a comment -

            maybe I misunderstood what the issue is: I thought the client was closing the connection because it was not receiving the pong notification in a timely manner.

            laurent Laurent Goujon added a comment - maybe I misunderstood what the issue is: I thought the client was closing the connection because it was not receiving the pong notification in a timely manner.

            I meant that the client is occupied and unable to send pings, so the server closes the connection.

            I do not see logic for ensuring that pongs are received every so often. So what you said is also not handled (but maybe that was the contract).

            sudheeshkatkam Sudheesh Katkam added a comment - I meant that the client is occupied and unable to send pings, so the server closes the connection. I do not see logic for ensuring that pongs are received every so often. So what you said is also not handled (but maybe that was the contract).
            laurent Laurent Goujon added a comment -

            maybe then having two separate services (one for sending, one for receiving) might make sense then. There are also some common locks you would have to figure out how to split...

            laurent Laurent Goujon added a comment - maybe then having two separate services (one for sending, one for receiving) might make sense then. There are also some common locks you would have to figure out how to split...

            People

              Unassigned Unassigned
              sudheeshkatkam Sudheesh Katkam
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: