[PIG-1842] Improve Scalability of the XMLLoader for large datasets such as wikipedia

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.7.0, 0.8.0, 0.9.0
• Fix Version/s: 0.8.1
• Component/s: impl
• Patch Info: Patch Available
• Assignee: Vivek Padmanabhan
• Reporter: Viraj Bhat
• Created: 04/Feb/11 02:20
• Updated: 25/Apr/11 21:27
• Resolved: 30/Mar/11 20:17

Description

The current XMLLoader for Pig, does not work well for large datasets such as the wikipedia dataset. Each mapper reads in the entire XML file resulting in extermely slow run times.

Viraj