Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5080 Create a memory-managed version of the External Sort operator
  3. DRILL-5066

External sort attempts to retry sv2 memory alloc, even if can never succeed

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.8.0
    • Fix Version/s: 1.11.0
    • Component/s: None
    • Labels:
      None

      Description

      The external sort contains rather complex code to allocate an sv2 in the method newSV2(). The code tries to allocate an sv2. If the allocation fails, the code attempts to spill (which is fine) and try again. If things still fail, the code waits 1 sec. and tries again. It will continue to wait up to a minute, doubling the wait time each cycle.

      Presumably, this is so that some other part of Drill will release memory. But, because of the way the allocator currently works, the allocation is limited by the limit set on the external sort's own allocator. This limit won't change by waiting.

      The loop only makes sense if the memory allocation failed because the external sort's allocator is not above its limit, but the parent can't provide memory.

      In practice, this scenario should not occur once the external sort is resource managed, so the retry code can simply be removed.

        Attachments

          Activity

            People

            • Assignee:
              paul-rogers Paul Rogers
              Reporter:
              paul-rogers Paul Rogers
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: