[IMPALA-11154] Idle Kudu daemons consume too much CPU

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 4.1.0
• Component/s: Infrastructure
• Epic Color: ghx-label-13
• Assignee: Zoltán Borók-Nagy
• Reporter: Zoltán Borók-Nagy
• Created: 25/Feb/22 13:43
• Updated: 09/Mar/22 08:21
• Resolved: 09/Mar/22 08:21

Description

I noticed that if I issue top in a machine that runs the minicluster, Kudu is always amongst the highest CPU consumers:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
29730 borokna+  20   0 1438712 101340  29256 S  27.8  0.3  17:43.76 kudu-tserver
29682 borokna+  20   0 1435780 100764  29360 S  16.7  0.3  16:24.49 kudu-tserver
29700 borokna+  20   0 1442964 104636  29452 S  16.7  0.3  18:18.77 kudu-tserver

The above is for a Kudu cluster that virtually doesn't do any work.

The flagfile looks like the following:

-rpc_bind_addresses=localhost:31202
-webserver_port=31302
-fs_wal_dir=/home/boroknagyz/Impala/testdata/cluster/cdh7/node-1/var/lib/kudu/ts/wal
-fs_data_dirs=/home/boroknagyz/Impala/testdata/cluster/cdh7/node-1/var/lib/kudu/ts/data
-log_dir=/home/boroknagyz/Impala/testdata/cluster/cdh7/node-1/var/log/kudu/ts

# The flags below require unsafe flags to be unlocked.
-unlock_unsafe_flags

# fsync is disabled for additional speed. Sometimes operations are slow on EC2/GCE test
# machines. Some data loss could occur if the system crashes before the OS has a chance
# to flush data to disk but that is acceptable for development purposes.
-never_fsync

# There is no need to require NTP-synchronized clock for tests where all the
# participating Kudu masters and tablet servers are run at the same node using the same
# local wallclock.
-time_source=system_unsync

# Enable Kudu transaction.
-enable_txn_system_client_init
-unlock_experimental_flags

Kudu version info is

kudu 67ba3cae45
revision 67ba3cae45143d083bdac119f45278ce0d2bfefb-dirty
build type RELEASE
built by None at 11 Jan 2022 01:08:09 UTC on 4cf70af93dc4

Issue Links

• is related to: KUDU-1973 Coalesce RPCs destined for the same server
• is related to: KUDU-3134 Adjust default value for --raft_heartbeat_interval