[HIVE-13303] spill to YARN directories, not tmp, when available

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Assignee: Sergey Shelukhin
• Reporter: Sergey Shelukhin
• Created: 17/Mar/16 22:08
• Updated: 14/Jul/16 06:11
• Resolved: 30/Mar/16 02:04

Description

RowContainer::setupWriter, HybridHashTableContainer::spillPartition, (KeyValueContainer|ObjectContainer)::setupOutput, VectorMapJoinRowBytesContainer::setupOutputFileStreams create files in tmp. Maybe some other code does it too, those are the ones I see on the execution path. When there are multiple YARN output directories and multiple tasks running on a machine, it's better to use the YARN directories. The only question is cleanup.

Issue Links

• is related to: HIVE-14167 Use work directories provided by Tez instead of directly using YARN local dirs