Details
Description
Since the Streaming Tab was added to the Spark UI in Spark 1.0, its has proven to be a defacto way to debug Spark Streaming applications at the time of development. However, there is much more we can do to help developers debug their streaming applications. Specifically we would like to show the details of individual batches and link them to the jobs that ran for that batch (similar to job details showing stages in the job). Furthermore we would also like show timelines to see trends over time for processing times data rates etc.
This is the design doc that shows the proposed new Streaming tab in the Spark UI.
https://docs.google.com/document/d/1-ZjvQ_2thWEQkTxRMHrVdnEI57XTi3wZEBUoqrrDg5c/edit?usp=sharing
Attachments
Issue Links
- is blocked by
-
SPARK-6766 StreamingListenerBatchSubmitted isn't sent and StreamingListenerBatchStarted.batchInfo.processingStartTime is a wrong value
- Resolved