Details
-
Sub-task
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
For Spark structured streaming, after the end of first micro-batch, the write client is closed and hence triggers closure of timeline service. But subsequent micro-batches do succeed though.
Currently, we explicitly override the marker type to be DIRECT in Spark structured streaming. We need to revisit the behavior of timeline service before we can enable timeline-server-based markers for Spark structured streaming.