Details
-
Improvement
-
Status: Open
-
P3
-
Resolution: Unresolved
-
None
-
None
Description
Users often ask how they can reason about the pipeline translation. The Flink UI display a confusingly large graph without any trace of the original Beam pipeline:
WordCount:
TFX:
Some aspects which make understanding these graphs hard:
- Users don't know how the Runner maps Beam to Flink concepts
- The UI is awfully slow / hangs when the pipeline is reasonable complex
- The operator names seem to use transform.getUniqueName() which doesn't generate readable name
- So called Chaining combines operators into a single operator which makes understanding which Beam concept belongs to which Flink concept even harder