[BEAM-6133] [SQL] Add support for TableMacro UDF

• Type: New Feature
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.10.0
• Component/s: dsl-sql
• Assignee: Gleb Kanterov
• Reporter: Gleb Kanterov
• Created: 27/Nov/18 13:29
• Updated: 24/Jul/20 19:54
• Resolved: 03/Jan/19 15:28

Description

Now we support only ScalarFunction UDFs. In Calcite, there are other kinds of UDFs. With TableMacro UDFs users can connect external data sources in a similar way as in TableProvider, but without specifying a schema, or enumerating a list of existing tables in advance.

An example use case is connecting external metadata service and querying range of partitions.

SELECT COUNT(*) FROM table(my_udf('dataset', start = '2017-01-01', end = '2018-01-01'))

Where the implementation of `my_udf` will connect to this service, get file locations for a range of partitions, and translate to PTransform reading it.

Issue Links

• links to: GitHub Pull Request #7141