Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5344

External sort priority queue copier fails with an empty batch

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.10.0
    • Fix Version/s: 1.11.0
    • Component/s: None
    • Labels:

      Description

      The external sort uses a "priority queue copier" to merge batches when spilling or when merging spilled batches.

      The code will fail with an IndexOutOfBoundsException if any record batch is empty. The reason is a faulty assumption in generated code:

        public void setup(...) {
          ...
            vector4.set(i, i, batchGroups.get(i).getNextIndex());
          ...
        }
      
        public int getNextIndex() {
          if (pointer == getRecordCount()) {
            return -1;
          }
          ...
        }
      

      The code to get the next index returns -1 when the "position" in a record batch is zero. The -1 position translates (when truncated) into 65535 which produces the index exception.

      The workaround has been to special case empty batches elsewhere in the code, apparently to avoid hitting this error.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                paul-rogers Paul Rogers
                Reporter:
                paul-rogers Paul Rogers
                Reviewer:
                Boaz Ben-Zvi
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: