[IMPALA-13479] Patch gperftools to allow max_total_thread_cache_bytes to exceed 1GB

• Type: Improvement
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: Impala 4.5.0
• Component/s: Infrastructure
• Epic Color: ghx-label-2
• Assignee: Unassigned
• Reporter: Joe McDonnell
• Created: 24/Oct/24 17:16
• Updated: 18/Nov/24 02:19

Description

gperftools limits max_total_thread_cache_bytes to 1GB here:

https://github.com/gperftools/gperftools/blob/gperftools-2.10/src/thread_cache.cc#L520-L523

void ThreadCache::set_overall_thread_cache_size(size_t new_size) {
  // Clip the value to a reasonable range
  if (new_size < kMinThreadCacheSize) new_size = kMinThreadCacheSize;
  if (new_size > (1<<30)) new_size = (1<<30);     // Limit to 1GB

I confirmed that setting --tcmalloc_max_total_thread_cache_bytes=2147483648 still results in a 1GB limit.

Sometimes, we would want a higher limit for systems with a large amount of memory and CPUs. For example, some systems now have 1TB of memory and 96 CPUs. With high concurrency, there is high contention on tcmalloc locks on central data structures. Increasing the total thread cache size could avoid this, and a value higher than 1GB is still a small part of system memory.

We can patch our toolchain gperftools to allow a higher value (and notify gperftools community).