Details
-
Test
-
Status: Open
-
P3
-
Resolution: Unresolved
-
2.9.0
-
None
-
None
Description
The JobService API (beam_job_api.proto) allows for the possibility of never seeing messages or states with Get(State|Message)Stream. This is because the Get(State|Message)Stream calls need to have the job id which can only be obtained from the RunJobResponse. But in order to see all messages/states the streams need to be opened before the job starts.
This is fine in Dataflow as the preparation_id == job_id, but this is not true in Flink.
Fix is to modify the API to only keep a single id to be used between the preparation/run APIs.
Consumers of the API will have to be modified to meet the new semantics.
Dev list thread (https://lists.apache.org/thread.html/3ace7585278c0545185fa4bb8d6975283d5c48c097e1bb2c2e18b9a2@%3Cdev.beam.apache.org%3E)