[DRILL-5617] Spill file name collisions when spill file is on a shared file system

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11.0
• Fix Version/s: 1.12.0
• Component/s: Functions - Drill
• Assignee: Chun Chang
• Reporter: Chun Chang
• Created: 28/Jun/17 23:28
• Updated: 30/Aug/17 21:01
• Resolved: 17/Aug/17 03:34

Description

Spill location can be configured to be written on hdfs such as:

hashagg: {

1. The partitions divide the work inside the hashagg, to ease
2. handling spilling. This initial figure is tuned down when
3. memory is limited.
4. Setting this option to 1 disables spilling !
   num_partitions: 32,
   spill: { # The 2 options below override the common ones # they should be deprecated in the future directories : [ "/tmp/drill/spill" ], fs : "maprfs:///" }

   }

However, this could cause spill filename conflict since name convention does not contain node name.

Issue Links

• Is contained by: DRILL-5601 Rollup of External Sort memory management fixes