[FLINK-16670] Support User-Defined Metrics in Python UDF

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Resolved
• Fix Version/s: 1.11.0
• Component/s: API / Python
• Assignee: Hequn Cheng
• Reporter: Hequn Cheng
• Created: 19/Mar/20 06:13
• Updated: 20/Apr/20 06:06
• Resolved: 03/Apr/20 16:38

Description

This is the umbrella Jira for FLIP112, which intends to support User-Defined Metrics in Python UDF.

FLIP wiki page: https://cwiki.apache.org/confluence/display/FLINK/FLIP-112%3A+Support+User-Defined+Metrics+in++Python+UDF

FLIP-58 adds the support for Python UDFs, but user-defined metrics have not been supported yet. With metrics, users can report and monitor the UDF status to get a deeper understanding of the execution. In this FLIP, we want to support metrics for Python UDFs. In this FLIP we propose to:

• Support user-defined metrics including Counters, Gauges, Meters, Distributions in Python UDFs. (Note: Histogram is not supported in this FLIP, instead, Distributions is supported to report statistics about the distribution of value. See more in the Distribution section.)
• Support defining user scopes.
• Support defining user variables.

Issue Links

• is duplicated by: FLINK-14024 Support use-defined metrics in Python user-defined functions
• mentioned in: Page Loading...

Sub-Tasks

• Support define scopes and variables on Python metric group: Closed
• Support Counter, Gauge, Meter, Distribution metric type for Python UDF: Closed
• Support Counter, Gauge, Meter, Distribution metric type for Python UDTF: Closed
• Add documentation about how to use user-defined metrics for Python UDF: Closed