[IMPALA-10545] Tune data_cache_write_concurrency based on the type of IO device

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 4.0.0
• Fix Version/s: Impala 4.2.0
• Component/s: Backend
• Labels: ramp-up
• Epic Color: ghx-label-14
• Assignee: Michael Smith
• Reporter: Joe McDonnell
• Created: 24/Feb/21 20:24
• Updated: 03/Aug/22 16:07
• Resolved: 03/Aug/22 16:07

Description

The data cache limits concurrency writes to the cache to avoid overwhelming the underlying IO device. This is controlled by the data_cache_write_concurrency flags and defaults to 1. For SSDs, we should be able to increase this to allow more concurrent writes to the data cache. This would allow the data cache to warm up faster and stay more up to date.

One option is to detect the underlying IO device (similar to how we do this for other parts of Disk IO Mgr) and tune this parameter higher for SSDs.