Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3359

Register Statements and Param Substitution in Macros

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: parser
    • Labels:
      None
    • Release Note:
      Register statements and parameter substitutions can be used inside macro now.

      Description

      There are some gaps in the functionality of macros that I've made a patch to address. The goal is to provide everything you'd need to make reusable algorithms libraries.

      1. You can't register udfs inside a macro
      2. Paramater substitutions aren't done inside macros
      3. Resources (including macros) should not be redundantly acquired if they are already present.

      Rohini's patch https://issues.apache.org/jira/browse/PIG-3204 should address problem 3 where Pig reparses everything every time it reads a line, but there still would be a problem if two separate files import the same macro / udf file.

      To get this working, I moved methods for registering jars/udfs and param substitution from PigServer to PigContext so they can be accessed in QueryParserDriver which processes macros (QPD was already passed a PigContext reference). Is that ok?

        Attachments

        1. PIG-3359-v7.diff
          68 kB
          Jonathan Packer
        2. PIG-3359-v6.diff
          63 kB
          Jonathan Packer
        3. PIG-3359-v5.diff
          58 kB
          Jonathan Packer
        4. PIG-3359-v4.diff
          57 kB
          Jonathan Packer
        5. PIG-3359-v3-test-failures.txt
          3 kB
          Cheolsoo Park
        6. PIG-3359-v3.diff
          61 kB
          Jonathan Packer
        7. PIG-3359-v2.diff
          34 kB
          Jonathan Packer
        8. PIG-3359-v1.diff
          35 kB
          Jonathan Packer
        9. PIG-3359_test.tar.gz
          253 kB
          Jonathan Packer

          Issue Links

            Activity

              People

              • Assignee:
                jpacker Jonathan Packer
                Reporter:
                jpacker Jonathan Packer
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: