Uploaded image for project: 'Calcite'
  1. Calcite
  2. CALCITE-3122

Convert Pig Latin scripts into Calcite logical plans

    XMLWordPrintableJSON

Details

    Description

      We create an internal Calcite repo at LinkedIn and develop APIs to parse any Pig Latin scripts into Calcite logical plan. The code was tested in nearly ~1000 Pig scripts written at LinkedIn.

      Changes:
      1. piglet: main conversion code live there, include:

      • APIs to convert any Pig scripts into RelNode plans or SQL statements
      • Use Pig Grunt parser to parse Pig Latin scripts into Pig logical plan (DAGs)
      • Convert Pig schemas into RelDatatype
      • Traverse through Pig expression plan and convert Pig expressions into RexNodes
      • Map┬ásome basic Pig UDFs to Calcite SQL operators
      • Build Calcite UDFs for any other Pig UDFs, including UDFs written in both Java and Python
      • Traverse (DFS) through Pig logical plans to convert each Pig logical nodes to RelNodes
      • Have an optimizer rule to optimize Pig group/cogroup into Aggregate operators

      2. core:

      • Implement other RelNode in Rel2Sql so that Pig can be translated into SQL
      • Other minor changes in a few other classes to make Pig to Calcite works

      Attachments

        Issue Links

          Activity

            People

              julianhyde Julian Hyde
              khaitran Khai Tran
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 20m
                  1h 20m