Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5226 External Sort encountered an error while spilling to disk
  3. DRILL-5312

"Record batch sizer" does not include overhead for variable-sized vectors

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.10.0
    • Fix Version/s: 1.11.0
    • Component/s: None
    • Labels:
      None

      Description

      The new "record batch sizer" computes the actual data size of a record given a batch of vectors. For most purposes, the record width must include the overhead of the offset vectors for variable-sized vectors. The initial code drop included only the character data, but not the offset vector size when computing row width.

      Since the "managed" external sort relies on the computed row size to determine memory usage, the underestimation of row count width can cause an OOM under certain low-memory conditions.

        Attachments

          Activity

            People

            • Assignee:
              paul-rogers Paul Rogers
              Reporter:
              paul-rogers Paul Rogers
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: