[SPARK-24933] SinkProgress should report written rows

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.1
• Fix Version/s: 3.0.0
• Component/s: Structured Streaming
• Assignee: Vaclav Kosar
• Reporter: Vaclav Kosar
• Created: 26/Jul/18 12:12
• Updated: 17/Dec/18 19:54
• Resolved: 17/Dec/18 19:52

Description

SinkProgress should report similar properties like SourceProgress as long as they are available for given Sink. Count of written rows is metric availble for all Sinks. Since relevant progress information is with respect to commited rows, ideal object to carry this info is WriterCommitMessage. For brevity the implementation will focus only on Sinks with API V2 and on Micro Batch mode. Implemention for Continuous mode will be provided at later date.

Before

{"description":"org.apache.spark.sql.kafka010.KafkaSourceProvider@3c0bd317"}

After

{"description":"org.apache.spark.sql.kafka010.KafkaSourceProvider@3c0bd317","numOutputRows":5000}

Issue Links

• is depended upon by: SPARK-24647 Sink Should Return Writen Offsets For ProgressReporting
• is related to: SPARK-24863 Report offset lag as a custom metrics for Kafka structured streaming source
• is related to: SPARK-21313 ConsoleSink's string representation
• relates to: SPARK-24748 Support for reporting custom metrics via Streaming Query Progress
• links to: [Github] Pull Request #21919 (vackosar)
• links to: GitHub Pull Request #21919