[PIG-1842] Improve Scalability of the XMLLoader for large datasets such as wikipedia - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: 0.7.0, 0.8.0, 0.9.0
Fix Version/s: 0.8.1
Component/s: impl
Labels:
None

Patch Info:

Patch Available

Description

The current XMLLoader for Pig, does not work well for large datasets such as the wikipedia dataset. Each mapper reads in the entire XML file resulting in extermely slow run times.

Viraj

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

PIG-1842_1.patch
08/Feb/11 07:42
19 kB
Vivek Padmanabhan
PIG-1842_2.patch
09/Feb/11 10:34
16 kB
Vivek Padmanabhan
TEST-org.apache.pig.piggybank.test.storage.TestXMLLoader.txt
25/Feb/11 19:44
40 kB
Alan Gates

Activity

People

Assignee:: Vivek Padmanabhan

Reporter:: Viraj Bhat

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 04/Feb/11 02:20

Updated:: 25/Apr/11 21:27

Resolved:: 30/Mar/11 20:17