[BEAM-10852] Combine Python streaming load test is too slow on Flink

• Type: Bug
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: runner-flink, sdk-py-core, testing
• Assignee: Unassigned
• Reporter: Kamil Wasilewski
• Created: 03/Sep/20 13:38
• Updated: 04/Jun/22 17:13

Description

One of the Combine load test cases, which involves a global combiner and data stream of 200M elements, takes too long on Flink. Flink is able to process only a half of that data stream within 1 hour, which is too long for a Jenkins job.

Job's definition: https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_LoadTests_Combine_Flink_Python.groovy#L36

Test pipeline: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/testing/load_tests/combine_test.py

Issue Links

• is related to: BEAM-10672 Create Streaming Combine Python Load Test Jenkins Job