Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5080 Create a memory-managed version of the External Sort operator
  3. DRILL-5055

External Sort does not delete spill file if error occurs during close

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.8.0
    • Fix Version/s: 1.11.0
    • Component/s: None
    • Labels:
      None

      Description

      DRILL-3898 recently fixed a case in which disk space was exhausted during a spill event for the external sort. In this case, the call to close failed because close attempted to flush remaining buffered data, but that also failed due to out of space.

      While the fix works, the fix causes the partially-completed spill file to be left on disk. Consider this code in BatchGroup.close( )

          if (outputStream != null) {
            outputStream.close();
          }
          ...
          if (fs != null && fs.exists(path)) {
            fs.delete(path, false);
          }
      

      Notice that, if the output stream close fails, the spill file is not deleted. The fix is to put the delete in a finally block so that it is always deleted.

        Attachments

          Activity

            People

            • Assignee:
              paul-rogers Paul Rogers
              Reporter:
              paul-rogers Paul Rogers
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: