[IMPALA-5520] TopN node does not reuse string memory

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Impala 2.10.0
• Component/s: Backend
• Labels: ramp-up resource-management
• Target Version: Product Backlog
• Epic Color: ghx-label-5
• Assignee: Bikramjeet Vig
• Reporter: Tim Armstrong
• Created: 15/Jun/17 21:31
• Updated: 24/Aug/17 16:36
• Resolved: 20/Jul/17 20:37

Description

In some cases TopN will use excessive memory. E.g. if you have a large number of input rows containing strings sorted in reverse order, it will allocate memory for all of the strings and never free it.

We should either recycle the allocations or periodically re-materialise and garbage collect the old allocations

There is a TODO in the code already.

      // TODO: DeepCopy() will allocate new buffers for the string data. This needs
      // to be fixed to use a freelist