Details

      Description

      Impala relies on Thrift 0.9.X for its RPC implementation. This is a venerable release and is showing its age. Problems include:

      • Low-quality SASL support (that we implemented ourselves)
      • Lack of high-quality nonblocking server (TNonBlockingServer is ok, but doesn't work with SASL, making it a non-starter for us. It is also very hard to provide support for sessions with TNonBlockingServer).
      • Lack of 0-copy native bytes type, meaning that large data structures are inefficient to send
      • Lack of support for async. server implementation, so expensive RPCs can consume threads that could have been used by cheap ones.

      Both Kudu and fbthrift have RPC implementations that address some or all of these shortcomings. We should evaluate both and commit to moving our RPC stack to one or the other.

        Attachments

          Issue Links

          1.
          Add Protobuf 2.6.1 to toolchain and as a build dependency Sub-task Resolved Henry Robinson
          2.
          Add libev 4.2.0 to toolchain and as a build dependency Sub-task Resolved Henry Robinson
          3.
          Add crcutil to toolchain Sub-task Resolved Henry Robinson
          4.
          Add Kudu's RPC, util and security libraries Sub-task Resolved Henry Robinson
          5.
          Add Kerberos minicluster test framework Sub-task Resolved Sailesh Mukil
          6.
          Add RpcMgr to interface between Impala and KRPC library Sub-task Resolved Michael Ho
          7.
          Upgrade gutil to recent Kudu version Sub-task Resolved Henry Robinson
          8.
          Port datastream portions of ImpalaInternalService to KRPC Sub-task Resolved Michael Ho
          9.
          Increase maximum KRPC message size Sub-task Resolved Joe McDonnell
          10.
          Replace kudu::ServicePool with one that uses Impala threads Sub-task Resolved Sailesh Mukil
          11.
          Refactor CreateImpalaServer() to allow it to be used in tests. Sub-task Resolved Sailesh Mukil
          12.
          Remove per-RPC DNS lookup Sub-task Resolved Michael Ho
          13.
          Upgrade LZ4 to recent version Sub-task Resolved Henry Robinson
          14.
          Enable KRPC Kerberos support in Impala Sub-task Resolved Sailesh Mukil
          15.
          Enable KRPC TLS in Impala Sub-task Resolved Sailesh Mukil
          16.
          Suppress kudu flags that aren't relevant to Impala Sub-task Resolved Sailesh Mukil
          17.
          Upgrade glog and gflags to most recent releases Sub-task Resolved Henry Robinson
          18.
          KRPC DCHECK hit when closing DataStreamRecvr Sub-task Resolved Henry Robinson
          19.
          Add Protobuf headers to Impala-lzo Sub-task Resolved Michael Ho
          20.
          Add krb5 to toolchain Sub-task Resolved Henry Robinson
          21.
          Openssl 1.0.0 shared library support for legacy platform Sub-task Resolved Michael Ho
          22.
          tcmalloc contention much higher with concurrency after KRPC patch Sub-task Resolved Mostafa Mokhtar
          23.
          Concurrent hung with lots of fragments blocked in KrpcDataStreamSender::Channel::WaitForRpc(std::unique_lock Sub-task Resolved Michael Ho
          24.
          RowBatch is serialized once per destination channel for Broadcast exchange Sub-task Resolved Michael Ho
          25.
          Address log spew originating from InboundCall::Respond() Sub-task Resolved Sailesh Mukil
          26.
          Bound memory usage of KRPC service queue Sub-task Resolved Michael Ho
          27.
          Crash in kudu::rpc::InboundCall::SerializeResponseBuffer inbound_call.cc:149 Sub-task Resolved Michael Ho
          28.
          KRPC w/ TLS doesn't work on remote clusters after rebase Sub-task Resolved Sailesh Mukil
          29.
          monotime.cc:200] Check failed: Initialized() Sub-task Resolved Michael Ho
          30.
          Disable rpc_default_keepalive_time_ms Sub-task Resolved Michael Ho
          31.
          Conflicting port 29000 with Sentry Sub-task Resolved Joe McDonnell
          32.
          Add source and destination hosts to TErrorCode::DATASTREAM_SENDER_TIMEOUT Sub-task Resolved Michael Ho
          33.
          Track RPC allocated memory in a memtracker Sub-task Resolved Lars Volker
          34.
          [observability] Add KRPC metrics to /rpcz and /metrics debug webpages Sub-task Resolved Lars Volker
          35.
          KerberosOnAndOff/RpcMgrKerberizedTest.MultipleServices failing Sub-task Resolved Sailesh Mukil
          36.
          Potential deadlock in KrpcDataStreamMgr Sub-task Resolved Sailesh Mukil
          37.
          Add a debug webpage to show fragment instances of a query Sub-task Resolved Lars Volker
          38.
          Add timeline information to fragment instances in profile Sub-task Resolved Lars Volker
          39.
          Monitor queue depth size for outgoing RPCs for Reactor threads Sub-task Resolved Sailesh Mukil
          40.
          Excessive synchronous logging in RpczStore::LogTrace causes severe slowdown for exchange operators spanning 2-3 minutes Sub-task Resolved Michael Ho
          41.
          Blocking in Service threads due to tcmalloc contention due to memory allocations for RowBatch::Deserialize Sub-task Resolved Michael Ho
          42.
          Allocate KrpcDataStreamRecvr RowBatch tuples from BufferPool Sub-task Resolved Michael Ho
          43.
          Use atomic operations for simple numeric metrics Sub-task Resolved Michael Ho
          44.
          Exchange node should correctly report peak memory in query profile and summary Sub-task Resolved Michael Ho
          45.
          Default rpc_negotiation_timeout_ms may cause queries to fail on large clusters Sub-task Resolved Sailesh Mukil
          46.
          Re-enable kerberized testing with KRPC Sub-task Resolved Sailesh Mukil
          47.
          Add flag to configure rpc_negotiation_timeout_ms and negotiation thread count in KRPC Sub-task Resolved Sailesh Mukil
          48.
          Allow tests to run with Krpc Sub-task Resolved Lars Volker
          49.
          test_exchange_delays does not work with KRPC enabled Sub-task Resolved Michael Ho
          50.
          test_rpc_exception does not work with KRPC enabled Sub-task Resolved Michael Ho
          51.
          rpc-mgr-kerberized-test fails on CentOS 6.4 Sub-task Resolved Sailesh Mukil
          52.
          Display number of rejected RPCs on the /metrics debug page Sub-task Resolved Lars Volker
          53.
          Check failed: consumption_->current_value() == 0 (126 vs. 0) KrpcDataStreamRecvr Sub-task Resolved Michael Ho
          54.
          Stress test with KRPC enabled shows inconsistent results for some queries Sub-task Resolved Michael Ho
          55.
          Add metric for Data Stream Service Queue memory consumption Sub-task Resolved Lars Volker
          56.
          Some COUNTER_ADD() in KrpcDataStreamRecvr may lead to use-after-free Sub-task Resolved Michael Ho
          57.
          Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c Sub-task Resolved Michael Ho
          58.
          KRPC : Data Stream Manager Deferred RPCs in memz page should be renamed Sub-task Resolved Lars Volker
          59.
          Improve profile in KrpcDataStreamRecvr and KrpcDataStreamSender Sub-task Resolved Michael Ho
          60.
          KRPC w/ kerberos fails on SLES11 Sub-task Resolved Sailesh Mukil
          61.
          Impala 2.12 & 3.0 Docs: KRPC Milestone 1 Sub-task Closed Alex Rodoni
          62.
          The combination use_kudu_kinit=false and use_krpc=true crashes Impalad Sub-task Resolved Michael Ho
          63.
          Catalog server's kerberos ticket gets deleted after 'ticket_lifetime' on SLES11 Sub-task Resolved Michael Ho

            Activity

              People

              • Assignee:
                kwho Michael Ho
                Reporter:
                henryr Henry Robinson
              • Votes:
                5 Vote for this issue
                Watchers:
                23 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: