Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5844

Fix management of FunctionContext "local" allocations.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Impala 2.11.0
    • Component/s: Backend
    • Labels:
      None

      Description

      FunctionContexts associated with expressions make two kind of allocations with very different lifecycles. One type of allocation is owned and managed by the expression, while the other "local" allocation is implicitly transferred to the Impala daemon after control flow returns from the expression. Both are currently allocated from the same pool.

      RowBatches returned from plan nodes may reference variable-length data stored in local allocations so this memory should be attached to the RowBatches.

      One approach here is:

      • Separate local and other allocations to allocate from different MemPools.
      • Manage local allocations in bulk by clearing, freeing, or transferring data from that MemPool, similar to other memory that would be allocated from a MemPool.

      I think there are some potential wrinkles here and work to set up correct MemPools for all places that use expressions but I think the high-level approach should work.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tarmstrong Tim Armstrong
                Reporter:
                tarmstrong Tim Armstrong
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: