Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-1717

Reduce backend TCP connections per-node to be independent of the number of backends

    XMLWordPrintableJSON

    Details

      Description

      We currently use up to |nodes|*|fragments| TCP connections per backend. Eventually, we should use a fixed number of connections over which all data stream sending can be multiplexed, but we will probably need to move away from Thrift before that happens, because we will need non-blocking IO to avoid deadlocks.

      In the meantime, we can use a fixed-size thread-pool per fragment, and at least only be linear in the number of fragments.

        Attachments

          Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-2567 KRPC milestone 1

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          is related to

          New Feature - A new feature of the product, which has yet to be developed. IMPALA-1718 Impala should use a non-blocking RPC stack for backend comms.

          • Major - Major loss of function.
          • Resolved

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                henryr Henry Robinson
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: