[SPARK-34270] Combine StateStoreMetrics should not override StateStoreCustomMetric

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.4, 2.4.7, 3.0.1, 3.1.2, 3.2.0
• Fix Version/s: 2.4.8, 3.0.2, 3.1.1
• Component/s: Structured Streaming
• Assignee: L. C. Hsieh
• Reporter: L. C. Hsieh
• Created: 27/Jan/21 21:27
• Updated: 30/Jan/21 05:21
• Resolved: 30/Jan/21 05:21

Description

For stateful join in structured streaming, we need to combine StateStoreMetrics from both left and right side. Currently we simply take arbitrary one from custom metrics with same name from left and right. By doing this we miss half of metric number.

Issue Links

• links to: [Github] Pull Request #31369 (viirya)