[CALCITE-4864] Supports Polymorphic Table function

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: core
• Assignee: Unassigned
• Reporter: Jing Zhang
• Created: 22/Oct/21 03:27
• Updated: 29/Jun/22 06:41

Description

Background

What makes table function to be polymorphic

A Polymorphic Table Function is introduced in SQL 2017 standard. It requires table function could take and return tables with row type not declared at design time.

In fact, table functions in Calcite already supports this function more or less. For example Tumble/Hop/Session Window table function could take a table parameter which row type is not declared at design time, and generate a return table which row type is not declared, either. However, we hadn't thought about how to support this feature systematically in the CALCITE.

Therefore, the current implementation misses some points and conflicts with the SQL standard, for example, for Session Window table function :

SELECT * FROM TABLE(
  SESSION(
    TABLE orders,
    DESCRIPTOR(rowtime),
    DESCRIPTOR(product),
    INTERVAL '20' MINUTE));

 
But, a desired way from SQL standard 2017 Polymorphic table functions is to use `PARTITION BY` clause to replace KEY DESCRIPTOR.

SELECT * FROM TABLE(
  SESSION(
    TABLE orders PARTITION BY product,
    DESCRIPTOR(rowtime),
    INTERVAL '20' MINUTE));

I propose to support polymorphic table function in a better way.

Row semantics and set semantics

Table function could have generic table parameters. And input table parameters are classified by three characteristics:

1. Input tables have either row semantics or set semantics)
   1. Row semantics means that the the result of the function is determined only by looking at the current row
   2. Set semantics means that the result of the function can be determined by looking at the current row and some state “summarized” from previously processed rows.  The table argument with set semantics can optionally be extended with either a PARTITION BY clause or an ORDER BY clause or both.
2. The second characteristic, which applies only to input tables with set semantics, is whether the function can generate a result row even if the input table is empty.
3. The third characteristic is whether the input table supports pass-through columns or not. Pass-through columns is a mechanism enabling the function to copy every column of an input row into columns of an output row.

Examples 

Table function contains a table parameter with row semantics

We often need to read a CSV file, generally, the first line of the file contains a list of column names, and subsequent lines of the file contain data. The data in general can be treated as a large VARCHAR. However, some of the fields may be numeric or datetime.
A PTF named CSVreader is designed to read a file of comma-separated values and interpret this file as a table.

CSVreader function has three parameters:

1. The first parameter, File, is the name of a file on the query author's system. This file must contain the comma-separated values that are to be converted to a table. The first line of the file contains the names of the resulting columns. Succeeding lines contain the data. Each line after the first will result in one row of output, with column names as determined by the first line of the input.
2. Floats is a PTF descriptor area, which should provide a list of the column names that are to be interpreted numerically. These columns will be output with the data type FLOAT.
3. Dates is a PTF descriptor area, which provides a list of the column names that are to be interpreted as datetimes. These columns will be output with the data type DATE.

How to use this table function in query?

For example, there is a csv file named abc.csv with the following contents:

docno,name,due_date,principle,interest
123,Mary,01/01/2014,234.56,345.67
234,Edgar,01/01/2014,654.32,543.21 

 
the query author may write a query such as the following:

SELECT *
FROM TABLE ( 
  CSVreader ( 
    File => 'abc.csv',
    Floats => DESCRIPTOR ("principle", "interest")
    Dates => DESCRIPTOR ("due_date")
  ) 
) AS S 

 

The result will be

docno	name	due_date	principle	interest
123	Mary	01/01/2014	234.56	345.67
234	Edgar	01/01/2014	654.32	543.21

 

Table function contains a table parameter with set semantics

TopN takes an input table that has been sorted on a numeric column. It copies the first n rows through to the output table. Any additional rows are summarized in a single output row in which the sort column has been summed and all other columns are null.
TopN function has two parameters:

1. The first parameter, Input, is the input table. This table has set semantics, meaning that the result depends on the set of data (since the last row is a summary row). The query author must order this input table on a single numeric column (syntax below).

1. The second parameter, Howmany, specifies how many input rows that the user wants to be copied into the output table; all rows after this will contribute to the final summary row in the output.

How to use this function in query?

For example, if the contents of source table orders are:

region	product	sales
East	A	1234.56
East	B	987.65
East	C	876.54
East	D	765.43
East	E	654.32
West	E	2345.67
West	D	2001.33
West	C	1357.99
West	B	975.35
West	A	864,22

 

the query author may write a query such as the following:

SELECT *
FROM TABLE(
  topn(
    Input => TABLE orders PARTITION BY region ORDER BY sales desc,
    Howmany => 3
  )
)

 

The result will be

region	product	sales
East	A	1234.56
East	B	987.65
East	C	876.54
West	E	2345.67
West	D	2001.33
West	C	1357.99

 

Issue Links

• blocks: FLINK-24002 Support count window with the window TVF
• blocks: FLINK-24024 Support session Window TVF
• blocks: FLINK-23582 Add documentation for session window
• relates to: CALCITE-4337 Support PARTITION BY clause in table function call
• relates to: CALCITE-4521 Support User Defined Table-valued Function

Sub-Tasks

• Allow table functions to be polymorphic: Closed
• Updates SqlWindowTableFunction because it's a Polymorphic Table Function: Open
• Updates DDL to support create a polymorphic table function: Open
• Expand table function document to introduce what makes a table function to be polymorphic, why we need this and give examples to explain how to use it: Open