[TAJO-992] Reduce number of hash shuffle output file.

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.0
• Component/s: Data Shuffle
• Assignee: Hyoungjun Kim
• Reporter: Hyoungjun Kim
• Created: 04/Aug/14 09:26
• Updated: 27/Aug/14 09:25
• Resolved: 27/Aug/14 09:25

Description

Currently Tajo creates too many intermediate files in the case of hash shuffle. A execution block(SubQuery) on a TajoWorker creates intermediate files as following rule:

1. intermediate files in a worker = # tasks / # workers * # partitions

This may cause 'too many file opens' error and makes it difficult to scale out. To solve this problem, We should reduce number of hash shuffle output file.