Apache Drill
  1. Apache Drill
  2. DRILL-336

Extend off heap memory manager to support growing vectors

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0.0-rc1
    • Fix Version/s: 1.0.0-rc1
    • Labels:

      Description

      Add the following interfaces to the bufferl memory manager:
      ptr = alloc(size, min, max) - allocate a block with given size, but
      with a total capacity between min+max.
      trim(ptr) - free up extra capacity, so current size is the capacity

      Rationale: It isn't always possible to anticipate the size of a vector. When creating a new vector, one strategy is to over-allocate the vector and then trim the size once the vector is complete. These routines allow us to implement this strategy.

        Activity

        Hide
        John Morris added a comment -

        Here is a revised API for extending the memory manager. The new API makes use of the existing interfaces, although with a slightly different interpretation.

        ByteBuffer buf = allocator.newDirectBuffer(min, max);
        Allocate a direct buffer with capacity between min and max.
        Formerly, max was an upper bound on resizing the buffer.
        Now, max is an opportunistic value - give us that much if convenient.

        buf.capacity(newSize);
        As before, resize the buffer to the new size and preserve the old data.
        The new version will

        • efficiently "trim" a buffer to a smaller size for "normal" allocations.
        • throw a "TooMuchCopying" exception when resizing requires
          copying more than a page of data.
        • Correctly resize the buffer, even if the exception is thrown.

        The "TooMuchCopying" exception is more a warning than an error. It signals an inefficient use of the capacity(newSize) method. Whether the exception is thrown or not, the buffer will be resized correctly.

        Show
        John Morris added a comment - Here is a revised API for extending the memory manager. The new API makes use of the existing interfaces, although with a slightly different interpretation. ByteBuffer buf = allocator.newDirectBuffer(min, max); Allocate a direct buffer with capacity between min and max. Formerly, max was an upper bound on resizing the buffer. Now, max is an opportunistic value - give us that much if convenient. buf.capacity(newSize); As before, resize the buffer to the new size and preserve the old data. The new version will efficiently "trim" a buffer to a smaller size for "normal" allocations. throw a "TooMuchCopying" exception when resizing requires copying more than a page of data. Correctly resize the buffer, even if the exception is thrown. The "TooMuchCopying" exception is more a warning than an error. It signals an inefficient use of the capacity(newSize) method. Whether the exception is thrown or not, the buffer will be resized correctly.

          People

          • Assignee:
            Unassigned
            Reporter:
            John Morris
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:

              Development