Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-34270

Combine StateStoreMetrics should not override StateStoreCustomMetric

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.4, 2.4.7, 3.0.1, 3.1.2, 3.2.0
    • Fix Version/s: 2.4.8, 3.0.2, 3.1.1
    • Component/s: Structured Streaming
    • Labels:
      None

      Description

      For stateful join in structured streaming, we need to combine StateStoreMetrics from both left and right side. Currently we simply take arbitrary one from custom metrics with same name from left and right. By doing this we miss half of metric number.

        Attachments

          Activity

            People

            • Assignee:
              viirya L. C. Hsieh
              Reporter:
              viirya L. C. Hsieh
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: