[SPARK-21988] Add default stats to StreamingRelation and StreamingExecutionRelation

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: Structured Streaming
• Assignee: Jose Torres
• Reporter: Jose Torres
• Created: 12/Sep/17 21:10
• Updated: 11/Oct/17 20:52
• Resolved: 11/Oct/17 20:51

Description

StreamingExecutionRelation currently doesn't implement stats.

This makes some sense, but unfortunately the LeafNode contract requires that nodes which survive analysis implement stats, and StreamingExecutionRelation can indeed survive analysis when running explain() on a streaming dataframe.

This value won't ever be used during execution, because StreamingExecutionRelation does not survive analysis on the execution path; it's replaced with each batch.

Issue Links

• links to: [Github] Pull Request #19212 (joseph-torres)
• links to: [Github] Pull Request #19465 (zsxwing)