[IMPALA-2550] Switch to per-query exec rpc

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.3.0
• Fix Version/s: Impala 2.9.0
• Component/s: Distributed Exec
• Labels: scalability
• Target Version: Impala 2.9.0
• Assignee: Marcel Kinard
• Reporter: Henry Robinson
• Created: 14/Oct/15 18:29
• Updated: 06/Apr/18 23:42
• Resolved: 13/May/17 16:45

Description

The execution-related rpcs in ImpalaInternalService (ExecPlanFragment, CancelPlanFragment) are currently fragment-instance oriented. This is very inefficient because a) starting up a single query involves lots of rpcs, b) it makes it hard to share query-wide data structures such as the descriptor table.

Instead, let's move to a per-query focus and replace the existing "execute" and "cancel" rpcs with per-query equivalents.

Issue Links

• breaks: IMPALA-5598 ExecQueryFInstances RPC recv side timeouts (observed in stress test in insecure+release build)
• breaks: IMPALA-5576 Wrong Cancel() in QueryState::ReportExecStatusAux() can lead to coordinator hang
• contains: IMPALA-6274 Impala daemon crashing in Coordinator::Wait() or some of its callees
• Dependency: IMPALA-3627 Clean up Rpc structures in ImpalaInternalService.thrift
• is blocked by: IMPALA-4014 Introduce query-wide execution state.
• is blocked by: IMPALA-1599 Improve query start-up time with many fragment instances
• is related to: IMPALA-4475 Compress ExecPlanFragment before shipping it to worker nodes to reduce network traffic
• relates to: IMPALA-2560 Large memory consumption on coordinator node when handling a large number of partitions due to redundancy in TDescriptorTable