Pig
  1. Pig
  2. PIG-3359

Register Statements and Param Substitution in Macros

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: parser
    • Labels:
      None
    • Release Note:
      Register statements and parameter substitutions can be used inside macro now.

      Description

      There are some gaps in the functionality of macros that I've made a patch to address. The goal is to provide everything you'd need to make reusable algorithms libraries.

      1. You can't register udfs inside a macro
      2. Paramater substitutions aren't done inside macros
      3. Resources (including macros) should not be redundantly acquired if they are already present.

      Rohini's patch https://issues.apache.org/jira/browse/PIG-3204 should address problem 3 where Pig reparses everything every time it reads a line, but there still would be a problem if two separate files import the same macro / udf file.

      To get this working, I moved methods for registering jars/udfs and param substitution from PigServer to PigContext so they can be accessed in QueryParserDriver which processes macros (QPD was already passed a PigContext reference). Is that ok?

      1. PIG-3359-v7.diff
        68 kB
        Jonathan Packer
      2. PIG-3359-v6.diff
        63 kB
        Jonathan Packer
      3. PIG-3359-v5.diff
        58 kB
        Jonathan Packer
      4. PIG-3359-v4.diff
        57 kB
        Jonathan Packer
      5. PIG-3359-v3-test-failures.txt
        3 kB
        Cheolsoo Park
      6. PIG-3359-v3.diff
        61 kB
        Jonathan Packer
      7. PIG-3359-v2.diff
        34 kB
        Jonathan Packer
      8. PIG-3359-v1.diff
        35 kB
        Jonathan Packer
      9. PIG-3359_test.tar.gz
        253 kB
        Jonathan Packer

        Activity

        Jonathan Packer created issue -
        Jonathan Packer made changes -
        Field Original Value New Value
        Attachment PIG-3359-v1.diff [ 12588915 ]
        Jonathan Packer made changes -
        Attachment PIG-3359_test.tar.gz [ 12588926 ]
        Jonathan Packer made changes -
        Description There are some gaps in the functionality of macros that I'll be making a patch to address. The goal is to provide everything you'd need to make reusable algorithms libraries.

        1. You can't register udfs inside a macro
        2. Paramater substitutions aren't done inside macros
        3. Resources (including macros) should not be redundantly acquired if they are already present.

        Rohini's patch https://issues.apache.org/jira/browse/PIG-3204 should address problem 3 where Pig reparses everything every time it reads a line, but there still would be a problem if two separate files import the same macro / udf file.
        There are some gaps in the functionality of macros that I've made a patch to address. The goal is to provide everything you'd need to make reusable algorithms libraries.

        1. You can't register udfs inside a macro
        2. Paramater substitutions aren't done inside macros
        3. Resources (including macros) should not be redundantly acquired if they are already present.

        Rohini's patch https://issues.apache.org/jira/browse/PIG-3204 should address problem 3 where Pig reparses everything every time it reads a line, but there still would be a problem if two separate files import the same macro / udf file.

        To get this working, I moved methods for registering jars/udfs and param substitution from PigServer to PigContext so they can be accessed in QueryParserDriver which processes macros (QPD was already passed a PigContext reference). Is that ok?
        Jonathan Packer made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Jonathan Packer made changes -
        Attachment PIG-3359-v2.diff [ 12591886 ]
        Jonathan Packer made changes -
        Attachment PIG-3359-v3.diff [ 12592848 ]
        Cheolsoo Park made changes -
        Attachment PIG-3359-v3-test-failures.txt [ 12593194 ]
        Jonathan Packer made changes -
        Attachment PIG-3359-v4.diff [ 12593279 ]
        Jonathan Packer made changes -
        Attachment PIG-3359-v5.diff [ 12593538 ]
        Jonathan Packer made changes -
        Attachment PIG-3359-v6.diff [ 12595038 ]
        Jonathan Packer made changes -
        Attachment PIG-3359-v7.diff [ 12596256 ]
        Cheolsoo Park made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Release Note Register statements and parameter substitutions can be used inside macro now.
        Fix Version/s 0.12 [ 12323380 ]
        Resolution Fixed [ 1 ]
        Cheolsoo Park made changes -
        Issue Type Bug [ 1 ] Improvement [ 4 ]
        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Jonathan Packer
            Reporter:
            Jonathan Packer
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development