[FLINK-36703] FLIP-440: User-defined SQL operators / ProcessTableFunction (PTF)

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: Table SQL / API, Table SQL / Planner, Table SQL / Runtime
• Assignee: Timo Walther
• Reporter: Timo Walther
• Created: 13/Nov/24 10:04
• Updated: 13/Nov/24 10:35

Description

Introduce a new kind of user-defined function (UDF) that enables implementing user-defined SQL operators: ProcessTableFunction (PTF)

The new UDF kind opens the Table & SQL API towards capabilities of the DataStream API while staying in the SQL ecosystem. And using all benefits of it.

PTFs look and feel familiar for both someone coming from the DataStream API world as well as the SQL world.

From SQL:

• Similar types and type inference as ScalarFunction, AggregateFunction, or TableFunction
• Registration in catalog, usage of inline/temporary UDF, built-in system functions in the future
• Usage in both SQL and Table API
• Very important: Standard-compliant syntax using Polymorphic Table Functions 

From DataStream API:

• Familiar naming like ProcessFunction
• Access to Map, List and Value state
• Ability to both keyBy() and connect() streams. Long-term also broadcast side functionality.
• Ability to deal with watermarks and dealing with time
• Support of query evolution (terminology as defined in FLIP-190)

 

Current implementation phases:
Phase 0: Single table input, no state, no timers, append-only
Phase 1: Value state
Phase 2: Time and timers
Phase 3: Changelog support
Phase 4: CoPartition for 2 inputs
Phase 5: Descriptors
Phase 6: Map and list state

Sub-Tasks

• Integrate PTF TypeInference into Calcite validator: Open
• Integrate PTF TypeInference into Table API: Open