[PIG-2898] Parallel execution of e2e tests

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.0
• Fix Version/s: 0.11, 0.12.0
• Component/s: e2e harness
• Labels: test
• Patch Info: Patch Available
• Assignee: Ivan A. Veselovsky
• Reporter: Andrey Klochkov
• Created: 30/Aug/12 18:42
• Updated: 22/Feb/13 04:53
• Resolved: 01/Nov/12 01:21

Description

Today it takes ~19 hours to run the full set of e2e tests in mapred mode. The bottleneck here is the client side, and per our observations it can help a lot if the e2e harness would be able to run tests in parallel threads.

We prototyped changes in e2e harness allowing to run tests in a configurable number of threads. Preliminary results show more than 6x reduction in execution time when using a small 3-nodes M/R cluster with modest configuration. Going to share a patch shortly.