[FLINK-11095] Table based ML Pipeline

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Component/s: Library / Machine Learning
• Assignee: Unassigned
• Reporter: Weihua Jiang
• Created: 07/Dec/18 08:08
• Updated: 10/May/19 02:48
• Resolved: 10/May/19 02:46

Description

As Table API will be the unified high level API for both batch and streaming, it is desired to have a table API based ML Pipeline definition. 

This new table based ML Pipeline shall:

1. support unified batch/stream ML/AI functionalities (train/inference).  
2. seamless integrated with flink Table based ecosystem. 
3. provide a base for further flink based AI platform/tooling support.
4. provide a base for further flink ML SQL integration. 

The initial design is here https://docs.google.com/document/d/1PLddLEMP_wn4xHwi6069f3vZL7LzkaP0MN9nAB63X90/edit?usp=sharing.

And based on this design, I made some initial implementation/prototyping. I will share the code later.

This is the umbrella JIRA. I will create corresponding sub-jira for each sub-task. 

Issue Links

• is duplicated by: FLINK-12470 FLIP39: Flink ML pipeline and ML libs

Sub-Tasks

• Create a new table based flink ML package: Closed
• Create a new set of table based ML Pipeline classes: Closed
• Create Table based optimizers: Closed
• Support pipeline stage type inference: Closed
• Create a new set of parameters: Closed
• Support pipeline import/export: Closed
• Support periodically update models when inferencing: Closed
• Support wrapping inference pipeline as a UDF function in SQL: Closed
• Port some flink.ml algorithms to table based: Closed