[IMPALA-5623] lag() on STRING cols may hold memory until query end

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.8.0
• Fix Version/s: Impala 2.10.0
• Component/s: Backend
• Target Version: Impala 2.10.0
• Epic Color: ghx-label-6
• Assignee: Matthew Jacobs
• Reporter: Matthew Jacobs
• Created: 06/Jul/17 21:28
• Updated: 11/Jul/17 15:41
• Resolved: 11/Jul/17 15:41

Description

IMPALA-4120 fixed an issue where lead/lag was potentially operating on memory that the UDA didn't own, resulting in potentially wrong results. As part of that fix, lead and lag started allocating memory in Init() which needs to be freed in Serialize() or Finalize(), but only lead was updated to free the memory. This memory is eventually freed when the fragment is torn down, but as a result of not freeing the memory in Serialize or Finalize, the memory may be allocated longer than necessary.

A warning is printed when this happens:

[localhost:21000] > select concat(' foo ', lag(string_col,1,NULL) over (partition by bool_col order by id)) from functional.alltypestiny order by id;
Query: select concat(' foo ', lag(string_col,1,NULL) over (partition by bool_col order by id)) from functional.alltypestiny order by id
Query submitted at: 2017-07-06 13:56:24 (Coordinator: http://mj-desktop.ca.cloudera.com:25000)
Query progress can be monitored at: http://mj-desktop.ca.cloudera.com:25000/query_plan?query_id=124dfe18a6cee76a:fafdea4000000000
+----------------------------------------------------------------------------------------+
| concat(' foo ', lag(string_col, 1, null) over (partition by bool_col order by id asc)) |
+----------------------------------------------------------------------------------------+
| NULL                                                                                   |
| NULL                                                                                   |
|  foo 0                                                                                 |
|  foo 1                                                                                 |
|  foo 0                                                                                 |
|  foo 1                                                                                 |
|  foo 0                                                                                 |
|  foo 1                                                                                 |
+----------------------------------------------------------------------------------------+
WARNINGS: UDF WARNING: Memory leaked via FunctionContext::Allocate() or FunctionContext::AllocateLocal()

Fetched 8 row(s) in 0.12s