[IMPALA-2567] KRPC milestone 1

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.2.4
• Fix Version/s: Impala 3.0, Impala 2.12.0
• Component/s: Distributed Exec
• Labels: scalability
• Target Version: Impala 2.12.0
• Assignee: Michael Ho
• Reporter: Henry Robinson
• Created: 19/Oct/15 22:48
• Updated: 06/Jul/20 15:44
• Resolved: 29/Mar/18 20:11

Description

Impala relies on Thrift 0.9.X for its RPC implementation. This is a venerable release and is showing its age. Problems include:

• Low-quality SASL support (that we implemented ourselves)
• Lack of high-quality nonblocking server (TNonBlockingServer is ok, but doesn't work with SASL, making it a non-starter for us. It is also very hard to provide support for sessions with TNonBlockingServer).
• Lack of 0-copy native bytes type, meaning that large data structures are inefficient to send
• Lack of support for async. server implementation, so expensive RPCs can consume threads that could have been used by cheap ones.

Both Kudu and fbthrift have RPC implementations that address some or all of these shortcomings. We should evaluate both and commit to moving our RPC stack to one or the other.

Issue Links

• contains: IMPALA-5577 Memory leak when looping a select, CTAS, and daemon crashes
• duplicates: IMPALA-1717 Reduce backend TCP connections per-node to be independent of the number of backends
• is blocked by: IMPALA-4905 Fragments always report insert status, even if not insert query
• is blocked by: IMPALA-5131 Known issues with KRPC (IMPALA-2567)
• is blocked by: KUDU-2305 Local variables can overflow when serializing a 2GB message
• is blocked by: KUDU-2228 Make Messenger options configurable
• is depended upon by: IMPALA-5441 Send larger row batches over the wire
• is depended upon by: IMPALA-5473 Make diagnosing network issues easier
• is duplicated by: IMPALA-4127 different fragment should not authenticate to the same node again in a short period of time
• is related to: IMPALA-9919 Bad Impala Performance after a period of time
• is related to: IMPALA-4317 Single Overloaded Impalad Causes Entire Cluster to Hang
• is related to: IMPALA-6720 impala deamon server 22000 connect refuse in cluster running time
• is related to: IMPALA-4069 Introduce startup option to create and cache backend connections on startup
• is related to: IMPALA-4127 different fragment should not authenticate to the same node again in a short period of time
• is related to: IMPALA-4952 Differentiate between resolved and unresolved IP addresses
• relates to: IMPALA-6238 Add source and destination hosts to TErrorCode::DATASTREAM_SENDER_TIMEOUT
• relates to: IMPALA-5865 Improve Impala execution scalability
• relates to: IMPALA-6025 Improve hang diagnostics
• relates to: KUDU-2011 Request-side sidecars cannot be safely destroyed on timeout
• relates to: KUDU-2110 RPC footer may be appended more than once
• relates to: IMPALA-2990 Coordinator should timeout and cancel queries with unresponsive / stuck executors
• relates to: KUDU-2086 Uneven assignment of connections to Reactor threads creates skew and limits transfer throughput
• relates to: KUDU-2192 KRPC should have a timer to close stuck connections
• relates to: IMPALA-5481 RowDescriptors should be shared, rather than copied
• relates to: IMPALA-6095 Microbenchmark for debugging KRPC issues
• supercedes: IMPALA-5199 Impala may hang on empty row batch exchange

Sub-Tasks

• Add Protobuf 2.6.1 to toolchain and as a build dependency: Resolved
• Add libev 4.2.0 to toolchain and as a build dependency: Resolved
• Add crcutil to toolchain: Resolved
• Add Kudu's RPC, util and security libraries: Resolved
• Add Kerberos minicluster test framework: Resolved
• Add RpcMgr to interface between Impala and KRPC library: Resolved
• Upgrade gutil to recent Kudu version: Resolved
• Port datastream portions of ImpalaInternalService to KRPC: Resolved
• Increase maximum KRPC message size: Resolved
• Replace kudu::ServicePool with one that uses Impala threads: Resolved
• Refactor CreateImpalaServer() to allow it to be used in tests.: Resolved
• Remove per-RPC DNS lookup: Resolved
• Upgrade LZ4 to recent version: Resolved
• Enable KRPC Kerberos support in Impala: Resolved
• Enable KRPC TLS in Impala: Resolved
• Suppress kudu flags that aren't relevant to Impala: Resolved
• Upgrade glog and gflags to most recent releases: Resolved
• KRPC DCHECK hit when closing DataStreamRecvr: Resolved
• Add Protobuf headers to Impala-lzo: Resolved
• Add krb5 to toolchain: Resolved
• Openssl 1.0.0 shared library support for legacy platform: Resolved
• tcmalloc contention much higher with concurrency after KRPC patch: Resolved
• Concurrent hung with lots of fragments blocked in KrpcDataStreamSender::Channel::WaitForRpc(std::unique_lock: Resolved
• RowBatch is serialized once per destination channel for Broadcast exchange: Resolved
• Address log spew originating from InboundCall::Respond(): Resolved
• Bound memory usage of KRPC service queue: Resolved
• Crash in kudu::rpc::InboundCall::SerializeResponseBuffer inbound_call.cc:149: Resolved
• KRPC w/ TLS doesn't work on remote clusters after rebase: Resolved
• monotime.cc:200] Check failed: Initialized(): Resolved
• Disable rpc_default_keepalive_time_ms: Resolved
• Conflicting port 29000 with Sentry: Resolved
• Add source and destination hosts to TErrorCode::DATASTREAM_SENDER_TIMEOUT: Resolved
• Track RPC allocated memory in a memtracker: Resolved
• [observability] Add KRPC metrics to /rpcz and /metrics debug webpages: Resolved
• KerberosOnAndOff/RpcMgrKerberizedTest.MultipleServices failing: Resolved
• Potential deadlock in KrpcDataStreamMgr: Resolved
• Add a debug webpage to show fragment instances of a query: Resolved
• Add timeline information to fragment instances in profile: Resolved
• Monitor queue depth size for outgoing RPCs for Reactor threads: Resolved
• Excessive synchronous logging in RpczStore::LogTrace causes severe slowdown for exchange operators spanning 2-3 minutes: Resolved
• Blocking in Service threads due to tcmalloc contention due to memory allocations for RowBatch::Deserialize: Resolved
• Allocate KrpcDataStreamRecvr RowBatch tuples from BufferPool: Resolved
• Use atomic operations for simple numeric metrics: Resolved
• Exchange node should correctly report peak memory in query profile and summary: Resolved
• Default rpc_negotiation_timeout_ms may cause queries to fail on large clusters: Resolved
• Re-enable kerberized testing with KRPC: Resolved
• Add flag to configure rpc_negotiation_timeout_ms and negotiation thread count in KRPC: Resolved
• Allow tests to run with Krpc: Resolved
• test_exchange_delays does not work with KRPC enabled: Resolved
• test_rpc_exception does not work with KRPC enabled: Resolved
• rpc-mgr-kerberized-test fails on CentOS 6.4: Resolved
• Display number of rejected RPCs on the /metrics debug page: Resolved
• Check failed: consumption_->current_value() == 0 (126 vs. 0) KrpcDataStreamRecvr: Resolved
• Stress test with KRPC enabled shows inconsistent results for some queries: Resolved
• Add metric for Data Stream Service Queue memory consumption: Resolved
• Some COUNTER_ADD() in KrpcDataStreamRecvr may lead to use-after-free: Resolved
• Network error: failed to write to TLS socket: error:1409F07F:SSL routines:SSL3_WRITE_PENDING:bad write retry:s3_pkt.c: Resolved
• KRPC : Data Stream Manager Deferred RPCs in memz page should be renamed: Resolved
• Improve profile in KrpcDataStreamRecvr and KrpcDataStreamSender: Resolved
• KRPC w/ kerberos fails on SLES11: Resolved
• Impala 2.12 & 3.0 Docs: KRPC Milestone 1: Closed
• The combination use_kudu_kinit=false and use_krpc=true crashes Impalad: Resolved
• Catalog server's kerberos ticket gets deleted after 'ticket_lifetime' on SLES11: Resolved