Pig
  1. Pig
  2. PIG-1793

Add macro expansion to Pig Latin

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      As production Pig scripts grow longer and longer, Pig Latin has a need to integrate standard programming techniques of separation and code sharing offered by functions and modules. A proposal of adding macro expansion to Pig Latin is posted here: http://wiki.apache.org/pig/TuringCompletePig

      Below is a brief summary of the proposed syntax (and examples):

      • Macro Definition

      The existing DEFINE keyword will be expanded to allow definitions of Pig macros.

      Syntax

      define <name> (<params>) returns <aliases> {
          <Pig Latin fragment>
      };
      

      Example

      define my_macro(A, sortkey) returns C {
          B = filter $A by my_filter(*);
          $C = order B by $sortkey;
      }
      
      • Macro Expansion

      Syntax

      <aliases> = <macro name> (<params>);
      

      Example: Use above macro in a Pig script:

      X = load 'foo' as (user, address, phone);
      Y = my_macro(X, user);
      store Y into 'bar';
      

      This script is expanded into the following Pig Latin statements:

      X = load 'foo' as (user, address, phone);
      macro_my_macro_B_1 = filter X by my_filter(*);
      Y = order macro_my_macro_B_1 by user;
      store Y into 'bar';
      

      Notes

      1. Any alias in the macro which isn't visible from outside will be prefixed with macro name and suffixed with instance id to avoid namespace collision.
      2. Macro expansion is not a complete replacement for function calls. Recursive expansions are not supported.

      • Macro Import

      The new IMPORT keyword can be used to add macros defined in another Pig Latin file.

      Syntax

      import <Pig Latin file name>;
      

      Example

      import my_macro.pig;
      

      Note: All macro names are in the global namespace.

      1. PIG-1793.patch
        85 kB
        Richard Ding
      2. PIG-1793_1.patch
        88 kB
        Richard Ding
      3. PIG-1793_2.patch
        19 kB
        Richard Ding

        Issue Links

          Activity

          Hide
          Richard Ding added a comment -

          Attaching the patch that implements the above proposed Pig syntax. The only change is the IMPORT statement which now requires the file name be a quoted string:

          import 'my_macro.pig';
          
          Show
          Richard Ding added a comment - Attaching the patch that implements the above proposed Pig syntax. The only change is the IMPORT statement which now requires the file name be a quoted string: import 'my_macro.pig';
          Hide
          Richard Ding added a comment -

          Unit tests pass. The output of test-patch:

               [exec] -1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     -1 javac.  The applied patch generated 973 javac compiler warnings (more than the trunk's current 962 warnings).
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     -1 release audit.  The applied patch generated 514 release audit warnings (more than the trunk's current 513 warnings).
          

          The release audit warning is html related.

          Show
          Richard Ding added a comment - Unit tests pass. The output of test-patch: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] -1 javac. The applied patch generated 973 javac compiler warnings (more than the trunk's current 962 warnings). [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 514 release audit warnings (more than the trunk's current 513 warnings). The release audit warning is html related.
          Hide
          Richard Ding added a comment -

          Attaching new patch that addresses the review comments (https://reviews.apache.org/r/400/).

          Show
          Richard Ding added a comment - Attaching new patch that addresses the review comments ( https://reviews.apache.org/r/400/ ).
          Hide
          Richard Ding added a comment -

          PIG-1793_1.patch is committed to trunk.

          Show
          Richard Ding added a comment - PIG-1793 _1.patch is committed to trunk.
          Hide
          Richard Ding added a comment -

          This patch (PIG-1793_2) adds support of macros invoking other macros:

          "Macros will be allowed to inline other macros. However, forward declarations will not be supported. Note that macro inlining is not a complete replacement for function calls, as it does not support recursive calls."

          Show
          Richard Ding added a comment - This patch ( PIG-1793 _2) adds support of macros invoking other macros: "Macros will be allowed to inline other macros. However, forward declarations will not be supported. Note that macro inlining is not a complete replacement for function calls, as it does not support recursive calls."
          Hide
          Richard Ding added a comment -

          Unit tests pass. The output of test-patch:

               [exec] -1 overall.  
               [exec] 
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec] 
               [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
               [exec] 
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec] 
               [exec]     -1 javac.  The applied patch generated 979 javac compiler warnings (more than the trunk's current 975 warnings).
               [exec] 
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec] 
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
          
          Show
          Richard Ding added a comment - Unit tests pass. The output of test-patch: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] -1 javac. The applied patch generated 979 javac compiler warnings (more than the trunk's current 975 warnings). [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Hide
          Richard Ding added a comment -

          See review comment here: https://reviews.apache.org/r/419/

          Show
          Richard Ding added a comment - See review comment here: https://reviews.apache.org/r/419/
          Hide
          Richard Ding added a comment -

          patch 2 committed to trunk.

          Show
          Richard Ding added a comment - patch 2 committed to trunk.

            People

            • Assignee:
              Richard Ding
              Reporter:
              Richard Ding
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development