Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5163

External sort on Mac creates a separate child process per spill via HDFS FS

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.8.0
    • Fix Version/s: 1.11.0
    • Component/s: None
    • Labels:
      None

      Description

      The external sort operator spills to disk. Spill files are created and written using the HDFS file system. For performance, HDFS uses native libraries to access the file system. These native libraries are not available on the Mac. As a result, some operations are implemented using a shower, Java-only path. One of these operations (need details) is implemented by forking a child process.

      When run in a debugger on the Mac, the behavior shows up as the furious creation and deletion of threads to manage the child processes: one per spill. Because of this behavior, performance of external sort is slow. Of course, no production code uses Drill on a Mac, so this is more of a nuisance than a real bug, which is why it is marked as an improvement.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                paul-rogers Paul Rogers
                Reporter:
                paul-rogers Paul Rogers
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: