[IMPALA-1472] Possibly dangerous interaction between tcmalloc and spilling.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.1
• Fix Version/s: Impala 2.1
• Assignee: Nong Li
• Reporter: Alexander Behm
• Created: 15/Nov/14 07:38
• Updated: 16/Nov/14 02:10
• Resolved: 16/Nov/14 02:10

Description

It seems like the tcmalloc and our spilling operators may interact badly in some cases. Try running TPCH-Q21 repeatedly from the Impala shell. Eventually a run will cause spilling, although there should be enough memory to run it without spilling. You can observe the consumption in /memz going up steadily after each run (and never decreasing). This is somewhat expected due to tcmalloc, but my understanding is that the consumption should eventually go down - which it does not.

Not marking 2.0 as affected because this behavior may have been introduced in recent unreleased changes. Might be good to verify our tcmalloc behavior is as expected on 2.0