[IMPALA-4063] Make fragment instance reports per-query (or per-host) instead of per-fragment instance.

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.7.0
• Fix Version/s: Impala 3.2.0
• Component/s: Distributed Exec
• Labels: impala-scalability-sprint-08-13-2018 performance
• Target Version: Impala 2.13.0, Impala 3.2.0
• Assignee: Michael Ho
• Reporter: Sailesh Mukil
• Created: 01/Sep/16 23:28
• Updated: 23/Nov/18 19:42
• Resolved: 06/Nov/18 01:11

Description

Currently we send a report per-fragment instance to the coordinator every 5 seconds (by default; modifiable via query option 'status_report_interval').

For queries with a large number of fragment instances, this generates tremendous amounts of network traffic to the coordinator, which will only be aggravated with higher a DOP.

We should instead queue per-fragment instance reports and send out a per-query report to the coordinator instead.

For code references, see:
PlanFragmentExecutor:: ReportProfile()
PlanFragmentExecutor:: SendReport()
FragmentExecState:: ReportStatusCb()

Issue Links

• blocks: IMPALA-4127 different fragment should not authenticate to the same node again in a short period of time
• blocks: IMPALA-4400 Aggregate runtime filters locally
• is blocked by: IMPALA-4014 Introduce query-wide execution state.
• is depended upon by: IMPALA-341 Remote profiles may be ignored by coordinator if query has a limit
• is related to: IMPALA-4555 Don't cancel query for failed ReportExecStatus (done=false) RPC
• is related to: IMPALA-4459 Consider making ReportExecStatus() RPC execute asynchronously
• relates to: IMPALA-2990 Coordinator should timeout and cancel queries with unresponsive / stuck executors
• requires: IMPALA-7163 Implement a state machine for the QueryState class