Details
-
Improvement
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
None
Description
Flink should run pipelines using the full portability API as currently defined:
https://s.apache.org/beam-fn-api
https://s.apache.org/beam-runner-api
https://s.apache.org/beam-job-api
https://s.apache.org/beam-fn-api-container-contract
This issue tracks its adoption of the portability framework. New Fn API and other features will be tracked separately.
Attachments
Issue Links
- Blocked
-
BEAM-4778 Less wasteful ArtifactStagingService
- Open
- depends upon
-
BEAM-2876 Add provision api proto
- Resolved
-
BEAM-3883 Python SDK stages artifacts when talking to job server
- Resolved
-
BEAM-4271 Executable stages allow side input coders to be set and/or queried
- Open
-
BEAM-4286 Pooled artifact source
- Open
-
BEAM-3792 Python submits portable pipelines to the Flink-served endpoint.
- Resolved
-
BEAM-3970 Java SDK harness supports window_into
- Resolved
-
BEAM-3972 Flink runner translates batch pipelines directly by proto
- Resolved
-
BEAM-3976 SdkHarnessClient is thread-safe
- Resolved
-
BEAM-4176 Java: Portable batch runner passes all ValidatesRunner tests that non-portable runner passes
- Resolved
-
BEAM-4285 Flink batch state request handler
- Resolved
-
BEAM-4523 Implement Flink batch ExecutableStage context
- Resolved
-
BEAM-5110 Reconile Flink JVM singleton management with deployment
- Resolved
-
BEAM-4069 Empty pipeline options can be gracefully serialized/deserialized
- Resolved
-
BEAM-4146 Python SDK sets environment in portable pipelines
- Resolved
-
BEAM-4149 SDK Harness should populate worker id in RPC headers
- Resolved
-
BEAM-4231 Runner utility for Coder instantiation
- Resolved
-
BEAM-4404 Fix docker run arguments
- Triage Needed
-
BEAM-4063 Flink runner supports cluster-wide artifact deployments through the Distributed Cache
- Open
-
BEAM-4068 Consistent option specification between SDKs and runners by URN
- Open
-
BEAM-2588 Portable Flink Runner Job API
- Resolved
-
BEAM-2897 Configurable container/process management
- Resolved
-
BEAM-3327 Add abstractions to manage Environment Instance lifecycles.
- Resolved
-
BEAM-4147 Abstractions for artifact delivery via arbitrary storage backends
- Resolved
-
BEAM-4615 Flink job server driver wrapper
- Resolved
-
BEAM-4056 Identify Side Inputs by PTransform ID and local name
- Resolved
-
BEAM-4067 Java: FlinkPortableTestRunner: runs portably via self-started local Flink
- Resolved
-
BEAM-4071 Java: PortableRunner - p.run() via given JobService
- Resolved
-
BEAM-4130 Portable Flink runner JobService entry point in a Docker container
- Resolved
-
BEAM-4131 Python SDK harness container image contains SDK and dependencies
- Resolved
-
BEAM-4148 Local server api descriptors contain urls that work on Mac and Linux
- Open
-
BEAM-2421 Migrate Apache Beam to use impulse primitive as the only root primitive
- Resolved
-
BEAM-2795 FlinkRunner: translate using SDK-agnostic means
- Resolved
-
BEAM-3914 'Unzip' flattens before performing fusion
- Resolved
-
BEAM-4267 Implement a reusable library that can run an ExecutableStage with a given Environment
- Resolved
-
BEAM-2880 Artifact server proxies
- Resolved
-
BEAM-4098 Handle WindowInto in the Java SDK Harness
- Resolved
- is blocked by
-
BEAM-5273 Local file system does not work as expected on Portability Framework with Docker
- Resolved
-
BEAM-5464 Portable beam hangs while running TFX preprocessing step on a distributed cluster
- Resolved
-
BEAM-5167 Use concurrency information from SDK Harness in Flink Portable Runner
- Open
-
BEAM-5187 Create a ProcessJobBundleFactory for non-dockerized SDK harness
- Open
-
BEAM-5284 Enable Java Portable Flink PostCommit Tests to Jenkins
- Open
-
BEAM-5283 Enable Python Portable Flink PostCommit Tests to Jenkins
- Resolved
- is depended upon by
-
BEAM-2896 Portability milestone: wordcount runs everywhere
- Triage Needed
- relates to
-
BEAM-4109 Support arbitrary artifact names in local file artifact staging service
- Open
-
BEAM-3994 Use typed sinks and sources for FnApiControlClientPoolService
- Resolved