Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2898

Parallel execution of e2e tests

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.10.0
    • 0.11, 0.12.0
    • e2e harness
    • Patch Available

    Description

      Today it takes ~19 hours to run the full set of e2e tests in mapred mode. The bottleneck here is the client side, and per our observations it can help a lot if the e2e harness would be able to run tests in parallel threads.

      We prototyped changes in e2e harness allowing to run tests in a configurable number of threads. Preliminary results show more than 6x reduction in execution time when using a small 3-nodes M/R cluster with modest configuration. Going to share a patch shortly.

      Attachments

        1. PIG-2898-trunk-3.patch
          27 kB
          Ilya Katsov
        2. PIG-2898-trunk-8.patch
          37 kB
          Ivan A. Veselovsky

        Activity

          People

            iveselovsky Ivan A. Veselovsky
            aklochkov Andrey Klochkov
            Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: