Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5623

lag() on STRING cols may hold memory until query end

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • Impala 2.8.0
    • Impala 2.10.0
    • Backend
    • None

    Description

      IMPALA-4120 fixed an issue where lead/lag was potentially operating on memory that the UDA didn't own, resulting in potentially wrong results. As part of that fix, lead and lag started allocating memory in Init() which needs to be freed in Serialize() or Finalize(), but only lead was updated to free the memory. This memory is eventually freed when the fragment is torn down, but as a result of not freeing the memory in Serialize or Finalize, the memory may be allocated longer than necessary.

      A warning is printed when this happens:

      [localhost:21000] > select concat(' foo ', lag(string_col,1,NULL) over (partition by bool_col order by id)) from functional.alltypestiny order by id;
      Query: select concat(' foo ', lag(string_col,1,NULL) over (partition by bool_col order by id)) from functional.alltypestiny order by id
      Query submitted at: 2017-07-06 13:56:24 (Coordinator: http://mj-desktop.ca.cloudera.com:25000)
      Query progress can be monitored at: http://mj-desktop.ca.cloudera.com:25000/query_plan?query_id=124dfe18a6cee76a:fafdea4000000000
      +----------------------------------------------------------------------------------------+
      | concat(' foo ', lag(string_col, 1, null) over (partition by bool_col order by id asc)) |
      +----------------------------------------------------------------------------------------+
      | NULL                                                                                   |
      | NULL                                                                                   |
      |  foo 0                                                                                 |
      |  foo 1                                                                                 |
      |  foo 0                                                                                 |
      |  foo 1                                                                                 |
      |  foo 0                                                                                 |
      |  foo 1                                                                                 |
      +----------------------------------------------------------------------------------------+
      WARNINGS: UDF WARNING: Memory leaked via FunctionContext::Allocate() or FunctionContext::AllocateLocal()
      
      Fetched 8 row(s) in 0.12s
      

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            mjacobs Matthew Jacobs
            mjacobs Matthew Jacobs
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment