Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21988

Add default stats to StreamingRelation and StreamingExecutionRelation

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.3.0
    • Component/s: Structured Streaming
    • Labels:
      None

      Description

      StreamingExecutionRelation currently doesn't implement stats.

      This makes some sense, but unfortunately the LeafNode contract requires that nodes which survive analysis implement stats, and StreamingExecutionRelation can indeed survive analysis when running explain() on a streaming dataframe.

      This value won't ever be used during execution, because StreamingExecutionRelation does not survive analysis on the execution path; it's replaced with each batch.

        Attachments

          Activity

            People

            • Assignee:
              joseph.torres Jose Torres
              Reporter:
              joseph.torres Jose Torres
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: