Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-3122

Convert Pig Latin scripts into Calcite logical plans

    XMLWordPrintableJSON

    Details

      Description

      We create an internal Calcite repo at LinkedIn and develop APIs to parse any Pig Latin scripts into Calcite logical plan. The code was tested in nearly ~1000 Pig scripts written at LinkedIn.

      Changes:
      1. piglet: main conversion code live there, include:

      • APIs to convert any Pig scripts into RelNode plans or SQL statements
      • Use Pig Grunt parser to parse Pig Latin scripts into Pig logical plan (DAGs)
      • Convert Pig schemas into RelDatatype
      • Traverse through Pig expression plan and convert Pig expressions into RexNodes
      • Map┬ásome basic Pig UDFs to Calcite SQL operators
      • Build Calcite UDFs for any other Pig UDFs, including UDFs written in both Java and Python
      • Traverse (DFS) through Pig logical plans to convert each Pig logical nodes to RelNodes
      • Have an optimizer rule to optimize Pig group/cogroup into Aggregate operators

      2. core:

      • Implement other RelNode in Rel2Sql so that Pig can be translated into SQL
      • Other minor changes in a few other classes to make Pig to Calcite works

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                julianhyde Julian Hyde
                Reporter:
                khaitran Khai Tran
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m