Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-8944

Python SDK harness performance degradation with UnboundedThreadPoolExecutor

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: P1
    • Resolution: Fixed
    • Affects Version/s: 2.18.0
    • Fix Version/s: 2.22.0
    • Component/s: sdk-py-harness
    • Labels:
      None

      Description

      We are seeing a performance degradation for python streaming word count load tests.

       

      After some investigation, it appears to be caused by swapping the original ThreadPoolExecutor to UnboundedThreadPoolExecutor in sdk worker. Suspicion is that python performance is worse with more threads on cpu-bounded tasks.

       

      A simple test for comparing the multiple thread pool executor performance:

       

      def test_performance(self):
         def run_perf(executor):
           total_number = 1000000
           q = queue.Queue()
      
          def task(number):
             hash(number)
             q.put(number + 200)
             return number
      
          t = time.time()
           count = 0
           for i in range(200):
             q.put(i)
      
          while count < total_number:
             executor.submit(task, q.get(block=True))
             count += 1
           print('%s uses %s' % (executor, time.time() - t))
      
         with UnboundedThreadPoolExecutor() as executor:
           run_perf(executor)
         with futures.ThreadPoolExecutor(max_workers=1) as executor:
           run_perf(executor)
         with futures.ThreadPoolExecutor(max_workers=12) as executor:
           run_perf(executor)
      

      Results:

      <apache_beam.utils.thread_pool_executor.UnboundedThreadPoolExecutor object at 0x7fab400dbe50> uses 268.160675049
      <concurrent.futures.thread.ThreadPoolExecutor object at 0x7fab40096290> uses 79.904583931
      <concurrent.futures.thread.ThreadPoolExecutor object at 0x7fab400dbe50> uses 191.179054976
      ```

      Profiling:
      UnboundedThreadPoolExecutor:

      1 Thread ThreadPoolExecutor:

      12 Threads ThreadPoolExecutor:

        Attachments

        1. profiling.png
          146 kB
          Yichi Zhang
        2. profiling_one_thread.png
          141 kB
          Yichi Zhang
        3. profiling_twelve_threads.png
          141 kB
          Yichi Zhang
        4. checkpoint-duration.png
          313 kB
          Maximilian Michels

          Issue Links

            Activity

              People

              • Assignee:
                lcwik Luke Cwik
                Reporter:
                yichi Yichi Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                11 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 8h 10m
                  8h 10m