[KUDU-3400] CompilationManager::RequestRowProjector consumed too much memory

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.12.0
• Component/s: codegen
• Assignee: Unassigned
• Reporter: Yingchun Lai
• Created: 15/Sep/22 04:11
• Updated: 14/Dec/22 07:19

Description

In one of our cluster, we find that CompilationManager::RequestRowProjector function consumed too much memory accidentally. Some situaction of this cluster:

1. some tables have more than 1000 columns, so the table schema may be very costly to copy
2. sometimes the tservers have memory pressure, and then do flush operations more frequently (to try to reduce memory consumed by MRS/DMS)

I catched a heap profile on a tserver, found out that CompilationManager::RequestRowProjector cost most memory when Schema copied, the source code:

 

CompilationTask(const Schema& base, const Schema& proj, CodeCache* cache,
                CodeGenerator* generator)
  : base_(base),
    proj_(proj),
    cache_(cache),
    generator_(generator) {} 

That is to say, Schemas (i.e. base and proj) are copied when construct CompilationTask objects.

The heap profile says that Schema consumed about 50GB memory, that really shock me, even though the Schema is large, but how can it consumed 50GB memory? I forget to `pstack` the process when it happend, maybe there are hundreds of thousands of CompilationManager::RequestRowProjector calls that time, but according to the code logic, it should not hang there for a long time?

if (!cached) {
  shared_ptr<CompilationTask> task(make_shared<CompilationTask>(
      *base_schema, *projection, &cache_, &generator_));
  WARN_NOT_OK_EVERY_N_SECS(pool_->Submit([task]() { task->Run(); }),
                  "RowProjector compilation request submit failed", 10);
  return false;
}