[PIG-96] It should be possible to spill big databags to HDFS

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.2.0
• Component/s: data
• Assignee: Unassigned
• Reporter: Pi Song
• Created: 06/Feb/08 11:58
• Updated: 25/Mar/10 00:12
• Resolved: 26/Jan/09 20:03

Description

Currently databags only get spilled to local disk which costs 2 disk io operations.If databags are too big, this is not efficient.
We should take advantage of HDFS so if the databag is too big (determined by DataBag.getMemorySize() > a big threshold), let's spill it to HDFS. Also read from HDFS in parallel when data is required.