Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2898

Parallel execution of e2e tests

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0
    • Fix Version/s: 0.11, 0.12.0
    • Component/s: e2e harness
    • Labels:
    • Patch Info:
      Patch Available

      Description

      Today it takes ~19 hours to run the full set of e2e tests in mapred mode. The bottleneck here is the client side, and per our observations it can help a lot if the e2e harness would be able to run tests in parallel threads.

      We prototyped changes in e2e harness allowing to run tests in a configurable number of threads. Preliminary results show more than 6x reduction in execution time when using a small 3-nodes M/R cluster with modest configuration. Going to share a patch shortly.

        Attachments

        1. PIG-2898-trunk-3.patch
          27 kB
          Ilya Katsov
        2. PIG-2898-trunk-8.patch
          37 kB
          Ivan A. Veselovsky

          Activity

            People

            • Assignee:
              iveselovsky Ivan A. Veselovsky
              Reporter:
              aklochkov Andrey Klochkov
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: