[ACCUMULO-375] Wikipedia Ingest needs more parallelism

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Assignee: Adam Fuchs
• Reporter: Adam Fuchs
• Created: 06/Feb/12 22:01
• Updated: 03/Apr/12 15:37
• Resolved: 03/Apr/12 15:37

Description

The wikipedia ingest Map job uses a derivative of the FileInputFormat, which launches one job per file. Given the partitioning strategy and workload distribution, it makes sense to launch multiple mappers per file. Each mapper can then take a chunk of the articles in the file using the same partitioning strategy as the assignment of row IDs.