Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-7736

SQL COPY: streaming model for network packets instead of request-response model

    XMLWordPrintableJSON

    Details

    • Type: Task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Problem
      Currently data transfer in COPY command is implemented as a series of request-responses. When request is received, it is parsed synchronously and passed to the streamer, then response is sent. This is not very efficient approach:

      1. We hardly could utilize long fat network channels efficiently as we spend a lot of time waiting for a very small response (ack).
      2. Parsing takes and adding data to the streamer takes time (especially if we reached streamer buffer limitations and are blocked waiting for responses from data nodes). During this period network is not utilized and file data is not transferred further.

      Solution
      Let's fix the problem iteratively as follows:

      1. Introduce asynchrony - when request is received, send the response immediately before data processing
      2. Then consider sending one ack for several requests instead of sending ack for every request
      3. When multiple simultaneous requests are enabled (previous point), consider moving data processing to separate stream, so that we can read data from the socket as fast as possible

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                vozerov Vladimir Ozerov
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: