[BEAM-5058] Python precommits should run E2E tests

• Type: Task
• Status: Resolved
• Priority: P2
• Resolution: Done
• Fix Version/s: Not applicable
• Component/s: sdk-py-core, testing
• Assignee: Mark Liu
• Reporter: Udi Meiri
• Created: 02/Aug/18 00:39
• Updated: 30/Sep/20 19:38
• Resolved: 14/Dec/18 00:15

Description

According to https://beam.apache.org/contribute/testing/ (which I'm working on), end-to-end tests should be run in precommit on each combination of {batch, streaming}x{SDK language}x{supported runner}.

At least 2 tests need to be added to Python's precommit: wordcount and wordcount_streaming on Dataflow, and possibly on other supported runners (direct runner and new runners plz).
These tests should be configured to run from a Gradle sub-project, so that they're run in parallel to the unit tests.

Example that parallelizes Java precommit integration tests: https://github.com/apache/beam/pull/5731

Issue Links

• links to: GitHub Pull Request #6707
• links to: GitHub Pull Request #7146
• links to: GitHub Pull Request #7163
• links to: GitHub Pull Request #7246