Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-7772

Stop using Perfkit Benchmarker tool in all tests

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: P2
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Not applicable
    • Component/s: testing
    • Labels:

      Description

      Devlist thread link

       

      Currently Python, IOIT and some Dataflow and Spark performance tests are relying on Perfkit Benchmarker tool. Due to the reasons discussed on the devlist it was decided to remove it from Beam's tests. 

      Problems that we face currently:

      1. Changes to Gradle tasks/build configuration in the Beam codebase have to be reflected in Perfkit code. This required PRs to Perfkit which can last and the tests break due to this sometimes (no change in Perfkit + change already there in beam = incompatibility). This is what happened in PR 8919 (above),
      2. Can't run in Python3 (depends on python 2 only library like functools32),
      3. Black box testing which hard to collect pipeline related metrics,
      4. Measurement of run time is inaccurate,
      5. It offers relatively small elasticity in comparison with eg. Jenkins tasks in terms of setting up the testing infrastructure (runners, databases). For example, if we'd like to setup Flink runner, and reuse it in consequent tests in one go, that would be impossible. We can easily do this in Jenkins.

      Tests that use Perfkit:

      1. IO integration tests,
      2. Python performance tests,
      3. beam_PerformanceTests_Dataflow (disabled),
      4. beam_PerformanceTests_Spark (failing constantly - looks not maintained).

       

        Attachments

          Issue Links

          There are no Sub-Tasks for this issue.

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                ŁukaszG Lukasz Gajowy
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 41.5h
                  41.5h