P3 Description
TestStream is a Test-Only primitive to help in verifying streaming SDK and Runner semantics.
It's a Known URN in the Beam Pipeline proto which Runners (like the Python Portable Runner) can implement, and SDKs can configure to achieve desired behavior.
This task is to implement Test Stream so it can be added to SDK and user test pipelines to simplify validating various SDK semantics.
This task is not to implement and support it in the Go Direct Runner at this time, though a separate Jira can be filed for that. At least direct runner is expected to fail, clearly saying it doesn't support the TestStream primitive. (Further improving the Direct Runner is in issue https://issues.apache.org/jira/browse/BEAM-11076)
Implementing further allows large sections of tests already implemented in Python and Java to be replicated in Go, to further improve confidence in the SDK implementation. Care needs to be taken for this though, many of those tests also validate the "runner" implementations of test stream itself, as Python and Java already have runner implementations.
To implement the Go SDK side of Test Stream look to the following:
The original TestStream Blog post describing it's overall purpose.
TestStreamPayload message for configuring test streams.
TEST_STREAM Urn for adding the test stream primitive to the pipeline graph.
Look at how Reshuffle was added to the Go SDK. While Reshuffle isn't a well known URN like TestStream is (and thus was more work) the same code end up needing to be modified to allow user side specification.
- A new TestStream edge needs to be added to the core/graph package.
- Handling translation of that edge into the TestStreamPayload needs to happen in the runtime/graphx package
- A user facing beam package entry point needs to be added with documentation so users can add TestStream to their pipelines.
- At that point, integration tests can begin to be added using the primitive, for supported runners.
Included in this work is various convenience helper functions or libraries to make using test stream simple to use for end users. This may include a new user facing package with various options depended on by the beam, graph, and graphx packages for configuration.
Like the rest of the SDK implementation, it's strongly recommended that Beam Pipeline Protos are handled in the graphx package to avoid overly coupling on a specific implementation of beam, should that change in the future.
Attachments
Issue Links
- links to
- mentioned in
-
Page Loading...
