[CALCITE-3122] Convert Pig Latin scripts into Calcite logical plans

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.21.0
• Component/s: core, piglet
• Labels: pull-request-available
• Assignee: Julian Hyde
• Reporter: Khai Tran
• Created: 07/Jun/19 18:20
• Updated: 15/Apr/21 13:11
• Resolved: 27/Aug/19 08:03

Description

We create an internal Calcite repo at LinkedIn and develop APIs to parse any Pig Latin scripts into Calcite logical plan. The code was tested in nearly ~1000 Pig scripts written at LinkedIn.

Changes:
1. piglet: main conversion code live there, include:

• APIs to convert any Pig scripts into RelNode plans or SQL statements
• Use Pig Grunt parser to parse Pig Latin scripts into Pig logical plan (DAGs)
• Convert Pig schemas into RelDatatype
• Traverse through Pig expression plan and convert Pig expressions into RexNodes
• Map some basic Pig UDFs to Calcite SQL operators
• Build Calcite UDFs for any other Pig UDFs, including UDFs written in both Java and Python
• Traverse (DFS) through Pig logical plans to convert each Pig logical nodes to RelNodes
• Have an optimizer rule to optimize Pig group/cogroup into Aggregate operators

2. core:

• Implement other RelNode in Rel2Sql so that Pig can be translated into SQL
• Other minor changes in a few other classes to make Pig to Calcite works

Issue Links

• relates to: CALCITE-3195 Handle UDF that throws checked exceptions in enumerable code generator
• relates to: CALCITE-3297 PigToSqlAggregateRule should be applied on multi-set projection to produce an optimal plan
• relates to: CALCITE-3307 PigRelExTest, PigRelOpTest and PigScriptTest fail on Windows
• relates to: CALCITE-3194 Convert Pig string patterns into SQL string patterns
• relates to: CALCITE-3193 Add RelBuilder.uncollect method, and interface UncollectFactory, to instantiate Uncollect
• links to: GitHub Pull Request #1265