Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-5617

Spill file name collisions when spill file is on a shared file system

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.11.0
    • Fix Version/s: 1.12.0
    • Component/s: Functions - Drill
    • Labels:
      None

      Description

      Spill location can be configured to be written on hdfs such as:

      hashagg: {

      1. The partitions divide the work inside the hashagg, to ease
      2. handling spilling. This initial figure is tuned down when
      3. memory is limited.
      4. Setting this option to 1 disables spilling !
        num_partitions: 32,
        spill: { # The 2 options below override the common ones # they should be deprecated in the future directories : [ "/tmp/drill/spill" ], fs : "maprfs:///" }

        }

      However, this could cause spill filename conflict since name convention does not contain node name.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                cchang@maprtech.com Chun Chang
                Reporter:
                cchang@maprtech.com Chun Chang
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: