Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-39134

Add custom metric of skipped null values for stream join operator

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • Structured Streaming
    • None

    Description

      When STATE_STORE_SKIP_NULLS_FOR_STREAM_STREAM_JOINS is enabled, Spark will skip null values in hash-based stream-stream joins. But currently we cannot track if this is happened and how many it is. For our customers, they have an interest to know the number of values skipped for better tracking purpose.

      Attachments

        Activity

          People

            viirya L. C. Hsieh
            viirya L. C. Hsieh
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: