[FLINK-12470] FLIP39: Flink ML pipeline and ML libs

• Type: New Feature
• Status: Closed
• Priority: Not a Priority
• Resolution: Won't Do
• Affects Version/s: 1.9.0
• Component/s: Library / Machine Learning
• Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned pull-request-available
• Assignee: Unassigned
• Reporter: Shaoxuan Wang
• Created: 10/May/19 02:23
• Updated: 19/Apr/23 01:39
• Resolved: 19/Apr/23 01:39

Description

This is the umbrella Jira for FLIP39, which intents to to enhance the scalability and the ease of use of Flink ML.

ML Discussion thread: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-39-Flink-ML-pipeline-and-ML-libs-td28633.html

Google Doc: (will convert it to an official confluence page very soon ) https://docs.google.com/document/d/1StObo1DLp8iiy0rbukx8kwAJb0BwDZrQrMWub3DzsEo

In machine learning, there are mainly two types of people. The first type is MLlib developer. They need a set of standard/well abstracted core ML APIs to implement the algorithms. Every ML algorithm is a certain concrete implementation on top of these APIs. The second type is MLlib users who utilize the existing/packaged MLlib to train or server a model.  It is pretty common that the entire training or inference is constructed by a sequence of transformation or algorithms. It is essential to provide a workflow/pipeline API for MLlib users such that they can easily combine multiple algorithms to describe the ML workflow/pipeline.

Current Flink has a set of ML core inferences, but they are built on top of dataset API. This does not quite align with the latest flink roadmap (TableAPI will become the first class citizen and primary API for analytics use cases, while dataset API will be gradually deprecated). Moreover, Flink at present does not have any interface that allows MLlib users to describe an ML workflow/pipeline, nor provides any approach to persist pipeline or model and reuse them in the future. To solve/improve these issues, in this FLIP we propose to:

• Provide a new set of ML core interface (on top of Flink TableAPI)
• Provide a ML pipeline interface (on top of Flink TableAPI)
• Provide the interfaces for parameters management and pipeline persistence
• All the above interfaces should facilitate any new ML algorithm. We will gradually add various standard ML algorithms on top of these new proposed interfaces to ensure their feasibility and scalability.

Issue Links

• duplicates: FLINK-11095 Table based ML Pipeline
• mentioned in: Page Loading...

Sub-Tasks

• Add the interface of ML pipeline and ML lib: Closed
• Remove the legacy flink-libraries/flink-ml: Closed
• Summarizer: summary statistics for Table: Closed
• ML common parameters: Closed
• Sparse and dense vector class, and dense matrix class with basic operations.: Resolved
• Add flink-ml-lib module: Closed
• Add more functionalities for ML Params and ParamInfo class: Closed
• Introduce FlinkML import/export framework: Closed
• Add the algorithm of Fast Fourier Transformation(FFT): Closed
• Add unary loss functions: Closed
• Add an implementation of pipeline's api: Resolved
• Add the Mapper and related classes for later algorithm implementations.: Resolved
• Add an util class to build result row and generate result schema.: Closed
• Add two utils for Table transformations.: Closed
• Add an implement of collector with the row type: Closed
• Add the utility class for the Table: Closed
• Add abstract classes for three typical scenarios of (Flat)Mapper.: Closed
• Add class for BinarizerMapper: Closed
• Add class for VectorAssemblerMapper: Closed
• Add class for VectorEleWiseProductMapper: Closed
• Add class for VectorInteractionMapper: Closed
• Add class of Vector Normalize Mapper: Closed
• Add class of Vector Size Hint Mapper: Closed
• Add class of Vector Slice Mapper: Closed
• Add class of Vector to Columns mapper: Closed
• Add Built-in vector types: Closed
• Add class for PolynomialExpansionMapper: Closed
• Add class for FeatureHasherMapper.: Closed
• Add the interface of ModelDataConverter, and several base classes that implement this interface.: Closed
• Add class for NLPConstant.: Closed
• Add class for DocHashTFVectorizerMapper.: Closed
• Add several base classes of summarizer.: Closed
• Add class for NGramMapper.: Closed
• Add class for RegexTokenizerMapper.: Closed
• Add class for TokenizerMapper.: Closed
• Add summarizer and summary for table.: Closed
• Add summarizer and summary for sparse vector and dense vector.: Closed
• Add class for DocHashIDFVectorizerModelMapper.: Closed
• Add class for DocCountVectorizerMapper.: Closed
• Add to BLAS a method that performs DenseMatrix and SparseVector multiplication.: Resolved
• Add the class for multivariate Gaussian Distribution.: Closed
• Add a wrapper class of a JSON library to provide the unified json format.: Closed
• Add an abstract class for mappers with rich model.: Closed
• Add the model mapper for Gaussian Mixture model.: Closed
• Add class for SqlOperators, and add sql operations to AlgoOperator, BatchOperator and StreamOperator.: Closed