Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5623

lag() on STRING cols may hold memory until query end

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.8.0
    • Fix Version/s: Impala 2.10.0
    • Component/s: Backend
    • Labels:
      None

      Description

      IMPALA-4120 fixed an issue where lead/lag was potentially operating on memory that the UDA didn't own, resulting in potentially wrong results. As part of that fix, lead and lag started allocating memory in Init() which needs to be freed in Serialize() or Finalize(), but only lead was updated to free the memory. This memory is eventually freed when the fragment is torn down, but as a result of not freeing the memory in Serialize or Finalize, the memory may be allocated longer than necessary.

      A warning is printed when this happens:

      [localhost:21000] > select concat(' foo ', lag(string_col,1,NULL) over (partition by bool_col order by id)) from functional.alltypestiny order by id;
      Query: select concat(' foo ', lag(string_col,1,NULL) over (partition by bool_col order by id)) from functional.alltypestiny order by id
      Query submitted at: 2017-07-06 13:56:24 (Coordinator: http://mj-desktop.ca.cloudera.com:25000)
      Query progress can be monitored at: http://mj-desktop.ca.cloudera.com:25000/query_plan?query_id=124dfe18a6cee76a:fafdea4000000000
      +----------------------------------------------------------------------------------------+
      | concat(' foo ', lag(string_col, 1, null) over (partition by bool_col order by id asc)) |
      +----------------------------------------------------------------------------------------+
      | NULL                                                                                   |
      | NULL                                                                                   |
      |  foo 0                                                                                 |
      |  foo 1                                                                                 |
      |  foo 0                                                                                 |
      |  foo 1                                                                                 |
      |  foo 0                                                                                 |
      |  foo 1                                                                                 |
      +----------------------------------------------------------------------------------------+
      WARNINGS: UDF WARNING: Memory leaked via FunctionContext::Allocate() or FunctionContext::AllocateLocal()
      
      Fetched 8 row(s) in 0.12s
      

        Attachments

          Activity

            People

            • Assignee:
              mjacobs Matthew Jacobs
              Reporter:
              mjacobs Matthew Jacobs
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: