[SPARK-30943] Show "batch ID" in tool tip string for Structured Streaming UI graphs

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 24/Feb/20 22:37
• Updated: 25/Feb/20 23:41
• Resolved: 25/Feb/20 23:31

Description

While we introduced the new UI for Structured Streaming, it was basically the port of Streaming UI to Structured Streaming.

That missed the major difference between twos, ID of the "batch". In Spark Streaming (DStream), timestamp is the key of the batch and you will need to find any relevant information with such timestamp.

In Structured Streaming, batchId is the key of the batch. It's not a simple difference because there're some known latency issues on periodical batches (at least in FileStreamSource and FileStreamSink) and it's bothering to find the right batchId if you only know about timestamp where the most of things are logged with the batchId in Structured Streaming world.

This issue tracks the efforts on showing batchId as main aspect of x axis, but only in tooltip string as changing x axis breaks coupling between Spark Streaming and Structured Streaming and lots of code should be duplicated and rewritten.

Issue Links

• links to: GitHub Pull Request #27687