[IMPALA-2567] KRPC milestone 1

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Impala 2.2.4
• Fix Version/s: Impala 3.0, Impala 2.12.0
• Component/s: Distributed Exec
• Labels: scalability
• Target Version: Impala 2.12.0
• Assignee: Michael Ho
• Reporter: Henry Robinson
• Created: 19/Oct/15 22:48
• Updated: 06/Jul/20 15:44
• Resolved: 29/Mar/18 20:11

Description

Impala relies on Thrift 0.9.X for its RPC implementation. This is a venerable release and is showing its age. Problems include:

• Low-quality SASL support (that we implemented ourselves)
• Lack of high-quality nonblocking server (TNonBlockingServer is ok, but doesn't work with SASL, making it a non-starter for us. It is also very hard to provide support for sessions with TNonBlockingServer).
• Lack of 0-copy native bytes type, meaning that large data structures are inefficient to send
• Lack of support for async. server implementation, so expensive RPCs can consume threads that could have been used by cheap ones.

Both Kudu and fbthrift have RPC implementations that address some or all of these shortcomings. We should evaluate both and commit to moving our RPC stack to one or the other.

Issue Links

• contains: IMPALA-5577 Memory leak when looping a select, CTAS, and daemon crashes
• duplicates: IMPALA-1717 Reduce backend TCP connections per-node to be independent of the number of backends
• is blocked by: IMPALA-4905 Fragments always report insert status, even if not insert query
• is blocked by: IMPALA-5131 Known issues with KRPC (IMPALA-2567)
• is blocked by: KUDU-2305 Local variables can overflow when serializing a 2GB message
• is blocked by: KUDU-2228 Make Messenger options configurable
• is depended upon by: IMPALA-5441 Send larger row batches over the wire
• is depended upon by: IMPALA-5473 Make diagnosing network issues easier
• is duplicated by: IMPALA-4127 different fragment should not authenticate to the same node again in a short period of time
• is related to: IMPALA-9919 Bad Impala Performance after a period of time
• is related to: IMPALA-4317 Single Overloaded Impalad Causes Entire Cluster to Hang
• is related to: IMPALA-6720 impala deamon server 22000 connect refuse in cluster running time
• is related to: IMPALA-4069 Introduce startup option to create and cache backend connections on startup
• is related to: IMPALA-4127 different fragment should not authenticate to the same node again in a short period of time
• is related to: IMPALA-4952 Differentiate between resolved and unresolved IP addresses
• relates to: IMPALA-6238 Add source and destination hosts to TErrorCode::DATASTREAM_SENDER_TIMEOUT
• relates to: IMPALA-5865 Improve Impala execution scalability
• relates to: IMPALA-6025 Improve hang diagnostics
• relates to: KUDU-2011 Request-side sidecars cannot be safely destroyed on timeout
• relates to: KUDU-2110 RPC footer may be appended more than once
• relates to: IMPALA-2990 Coordinator should timeout and cancel queries with unresponsive / stuck executors
• relates to: KUDU-2086 Uneven assignment of connections to Reactor threads creates skew and limits transfer throughput
• relates to: KUDU-2192 KRPC should have a timer to close stuck connections
• relates to: IMPALA-5481 RowDescriptors should be shared, rather than copied
• relates to: IMPALA-6095 Microbenchmark for debugging KRPC issues
• supercedes: IMPALA-5199 Impala may hang on empty row batch exchange