[BEAM-6335] GroupByKey uses data insertion pipeline in streaming tests

• Type: Sub-task
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: testing
• Assignee: Unassigned
• Reporter: Kasia Kucharczyk
• Created: 02/Jan/19 17:30
• Updated: 15/May/21 17:55
• Resolved: 15/May/21 17:55

Description

Uses prepared Java Data Insertion Pipeline to update GroupByKey in Python to load test streaming.

This task contains following steps:

1. Create GroupByKey streaming test that accepts bytes
2. To stop test after arrived messages it is required to add a matcher. The matcher should work on number of messages because in case of load testing it would be difficult to compare big load of bytes (also casting bytes to string to compare load would be difficult).
3. All data is generated by SyntheticDataPublisher.java, which is sending bytes produced by synthetic source to PubSub. PubSub is used as a streaming source for the Python test.

Issue Links

• is related to: BEAM-6201 Provide a "Data insertion pipeline" for Python suites
• relates to: BEAM-6397 Add KeyValue attribute to SyntheticDataPubSubPublisher
• links to: GitHub Pull Request #10029