[KUDU-2995] Tuning kudu write performance

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Information Provided
• Affects Version/s: 1.7.0
• Fix Version/s: n/a
• Component/s: client, java
• Labels: performance
• Environment: kudu 1.7.0-cdh5.16.2
• Assignee: Unassigned
• Reporter: wxmimperio
• Created: 11/Nov/19 04:10
• Updated: 12/Nov/19 06:25
• Resolved: 12/Nov/19 06:25

Description

Kudu version： kudu 1.7.0-cdh5.16.2

System memary pre node：256G

4 SSDs per machine：512G

Three Master nodes and three Tserver nodes.

// Master config
--fs_wal_dir=/mnt/disk1/kudu/var/wal
--fs_data_dirs=/mnt/disk1/kudu/var/data,/mnt/disk2/kudu/var/data,/mnt/disk3/kudu/var/data,/mnt/disk4/kudu/var/data
--fs_metadata_dir=/mnt/disk1/kudu/var/metadata
--log_dir=/mnt/disk1/kudu/var/logs
--master_addresses=xxxx
--maintenance_manager_num_threads=2
--block_cache_capacity_mb=6144
--memory_limit_hard_bytes=34359738368
--max_log_size=40

// Tserver config
--fs_wal_dir=/mnt/disk1/kudu/var/wal
--fs_data_dirs=/mnt/disk1/kudu/var/data,/mnt/disk2/kudu/var/data,/mnt/disk3/kudu/var/data,/mnt/disk4/kudu/var/data
--fs_metadata_dir=/mnt/disk1/kudu/var/metadata
--log_dir=/mnt/disk1/kudu/var/logs
--tserver_master_addrs=xxxx
--block_cache_capacity_mb=6144
--memory_limit_hard_bytes=34359738368
--max_log_size=40

// Table schema
// _key is UUID for each msg
// event_time is data time
// Schema has only 15 columns
// Single message does not exceed 100Bytes

HASH (_key) PARTITIONS 3,
RANGE (event_time) (
    PARTITION 2019-10-31T16:00:00.000000Z <= VALUES < 2019-11-30T16:00:00.000000Z
)

I write a project to write data to kudu.

Whether manual or automatic flush mode write speed is only 6MB/s.

I think SSD should be more than this speed, and the network and memory have not reached the bottleneck.

Is this the normal level of kudu writing? How to tuning?