[FLINK-12308] Support python language in Flink Table API

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.9.0
• Fix Version/s: 1.10.0
• Component/s: Table SQL / API
• Labels: pull-request-available
• Assignee: sunjincheng
• Reporter: sunjincheng
• Created: 24/Apr/19 05:08
• Updated: 09/Dec/19 06:30
• Resolved: 09/Dec/19 06:30

Description

At the Flink API level, we have DataStreamAPI/DataSetAPI/TableAPI&SQL, the Table API will become the first-class citizen. Table API is declarative, and can be automatically optimized, which is mentioned in the Flink mid-term roadmap by Stephan. So, first considering supporting Python at the Table level to cater to the current large number of analytics users. And Flink's goal for Python Table API as follows:

• Users can write Flink Table API job in Python, and should mirror Java / Scala Table API
• Users can submit Python Table API job in the following ways:
  • Submit a job with python script, integrate with `flink run`
  • Submit a job with python script by REST service
  • Submit a job in an interactive way, similar `scala-shell`
  • Local debug in IDE.
• Users can write custom functions(UDF, UDTF, UDAF)
• Pandas functions can be used in Flink Python Table API

A more detailed description can be found in FLIP-38.

For the API level, we make the following plan：

• The short-term:
  We may initially go with a simple approach to map the Python Table API to the Java Table API via Py4J.

• The long-term:
  We may need to create a Python API that follows the same structure as Flink's Table API that produces the language-independent DAG. (As Stephan already motioned on the mailing thread)

Sub-Tasks

• Add base python framework and Add Scan, Projection, and Filter operator support: Closed
• Add a basic test framework, just like the existing Java TableAPI, abstract some TestBase.: Closed
• Add simplicity support for submitting Python Table API job in CliFrontend, i.e. `flink run -py wordcount.py` can be work(with simple test).: Closed
• Add integrated Tox for ensuring compatibility with the python2/3 version: Closed
• Integrated Travis for Python Table API: Closed
• Add all table operators align Java Table API: Closed
• Support to define all kinds of types in Python API: Closed
• Adds from_elements in TableEnvironment: Closed
• Add FileSystem Connector with CSV format support in Python Table API: Closed
• Add all connector support align Java Table API: Closed
• Add custom check options for lint-python.sh: Closed
• Add a tool to check the user interface of Python Table API aligns with Java Table API.: Closed
• Move PythonGatewayServer into flink-clients: Closed
• Add deploy a Python Flink job and session cluster on Kubernetes support.: Reopened
• Reduce the test cost for Python API: Closed
• Add all format support align with the Java Table API: Closed
• Align Stream/BatchTableEnvironment with JAVA Table API: Closed
• Add TableSchema for Python Table API: Closed
• Align the Python data types with Java: Closed
• Enable the configuration of using blink planner: Closed
• Add an interactive shell for Python Table API: Closed
• Add windows support for PyFlink: Closed
• Add the Python catalog API: Closed
• Add the Python Table API Sphinx docs: Closed
• Adds Python Table API tutorial: Closed
• Adds a wiki page about setting up a Python Table API development environment: Closed
• Improves the Python word_count example to use the descriptor API: Closed
• Support user defined connectors/format: Closed
• Allow to specify directory in option -pyfs: Closed
• Add support to run a Python job-specific cluster on Kubernetes: Closed
• Correct the package name for python API: Closed
• Add support for build Python Docs in Buildbot: Closed
• Improves the performance of Python Table API test cases: Closed
• Improve the Python Table API docs by adding more examples: Closed
• Allows pyflink to be pip installed: Closed
• Release the PyFlink into PyPI: Closed
• Supported java UDFs in python API: Closed