Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2567 KRPC milestone 1
  3. IMPALA-6725

Impala 2.12 & 3.0 Docs: KRPC Milestone 1

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Won't Fix
    • Impala 3.0, Impala 2.12.0
    • None
    • Docs
    • None
    • ghx-label-3

    Description

      IMPALA-2567 replaces the RPC system for transmitting data between Impala backends with Kudu RPC which provides asynchronous RPC and connection multiplexing. Previously, Impala uses Thrift RPC and Impala will create a connection and a thread between each fragment instance pair. This leads to excessive number of connections and threads created when running complex queries on a moderate size cluster, causing crashes of Impala and inefficient CPU scheduling. After switching to KRPC, the number of threads responsible for RPC transportation will be fixed. There is only one pair of connection between any two hosts, one for each direction. Note that we still use the legacy thrift RPC system for some other RPCs between Impala backends so we may still exert some pressure on KDC in a cold cluster. We plan to eventually replace all RPCs between Impala backends with KRPC.

      This KRPC feature is enabled by default and it can be disabled by using the start-up flag --use_krpc=false.

      Attachments

        Activity

          People

            arodoni Alexandra Rodoni
            arodoni Alexandra Rodoni
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: