[IMPALA-11659] Server network util of Statestored is much higher than impala-3.4

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 4.1.0, Impala 4.1.1
• Fix Version/s: Impala 4.2.0
• Component/s: Perf Investigation
• Target Version: Impala 4.1.3
• Epic Color: ghx-label-10
• Assignee: Zhi Tang
• Reporter: Yuchen Fan
• Created: 13/Oct/22 12:36
• Updated: 07/Mar/24 00:09
• Resolved: 12/May/23 02:02

Description

We found that server network traffic of Statestored is rised immediatrly after upgraded to Impala-4.1. Using 'iftop' shows Statestored has aboud 6MB/s outbound network communication with every Impalad. With more than 250 nodes of Impalad, Statestored server has 10~15Gb/s(1~2GB/s) outbound(60x higher than it before upgrade) and about 90Mb/s(>10MB/s) inbound(10x higher than it before upgrade) network communication, which will occupy about 75% server network util. TCP packet snapshot shows detail of packet is 'Pool' and 'Stat' information in impala-request-queue topic. We found that value of key 'STAT:' contains per host stat of all Impalads. Related function is from https://issues.apache.org/jira/browse/IMPALA-8762. So initial update size of impala-request-queue is more than 4MB when there are more than 250 Impalads. If cluster has more Impalad, Statestored server may hang because of network blocking.

I think coordinators updating topic of per host statistics is just fine. All Impalads(including coordinators and executors) registering subscriber to topic of impala-request-queue makes Statestored need to broadcast all per node statistics to all Impalads.

Issue Links

• is caused by: IMPALA-8762 Track number of running queries on all backends in admission controller
• links to: Code Review