Pig
  1. Pig
  2. PIG-2898

Parallel execution of e2e tests

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.11, 0.12.0
    • Component/s: e2e harness
    • Labels:
    • Patch Info:
      Patch Available

      Description

      Today it takes ~19 hours to run the full set of e2e tests in mapred mode. The bottleneck here is the client side, and per our observations it can help a lot if the e2e harness would be able to run tests in parallel threads.

      We prototyped changes in e2e harness allowing to run tests in a configurable number of threads. Preliminary results show more than 6x reduction in execution time when using a small 3-nodes M/R cluster with modest configuration. Going to share a patch shortly.

      1. PIG-2898-trunk-8.patch
        37 kB
        Ivan A. Veselovsky
      2. PIG-2898-trunk-3.patch
        27 kB
        Ilya Katsov

        Activity

          People

          • Assignee:
            Ivan A. Veselovsky
            Reporter:
            Andrey Klochkov
          • Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development