Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.1.0
    • Component/s: None
    • Labels:
      None

      Description

      This feature has been requested by several users and would be very useful in conjunction with streaming. The feature would allow pig script to include parameters that are replaced at run time. For instance, if your script needs to run on a daily basis over the data of the previous day, you would be able to use the script and providing a date as a run-time parameter to it.

      Example:
      =======
      Pig script myscript.pig:

      A = load '/data/mydata/%date%';
      B = filter A by $0>'5';
      .....

      Pig command line:

      pig -param date='20080110' myscript.pig

      Proposed interface and implementation:

      Interface:
      =======
      (0) Substitution will be only supported with pig script files.
      (1) Parameters are specified on the command line via -param <param>=<val> construct. Multiple parameters can be specified. They are applied to the script in the order they are specified on the command line
      (2) Default values for the parameters can be specified within the script via decare statement:
      decare <param>=<value>
      (3) Withint the script the parameter will be enclosed in %%. % can be used te escape.

      Implementation:
      ============

      Use preprocessor to do the substitution. The preprocessor would be invoced by Main before grunt is instanciated and do the following:

      • create a new file in temp location
      • build a hash of parameters from command line and declare statement
      • for each line in the original script
        if this is a declare line, skip it
        else for each unescaped pattern %<identifie>% look for a match in the hash. Replace, if found. Write the line to the temp file.
      • pass the temp file to grunt.
      1. PIG-58_v1.patch
        96 kB
        Olga Natkovich
      2. PIG-58_v2
        97 kB
        Olga Natkovich
      3. PIG-58_v3.patch
        103 kB
        Olga Natkovich

        Activity

        Olga Natkovich created issue -
        Owen O'Malley made changes -
        Field Original Value New Value
        Workflow jira [ 12420655 ] no-reopen-closed, patch-avail [ 12425400 ]
        Olga Natkovich made changes -
        Attachment PIG-58_v1.patch [ 12379517 ]
        Olga Natkovich made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Olga Natkovich made changes -
        Attachment PIG-58_v2 [ 12379588 ]
        Olga Natkovich made changes -
        Attachment PIG-58_v3.patch [ 12379700 ]
        Olga Natkovich made changes -
        Resolution Fixed [ 1 ]
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Olga Natkovich made changes -
        Fix Version/s 0.1.0 [ 12312848 ]
        Alan Gates made changes -
        Assignee Olga Natkovich [ olgan ]
        Alan Gates made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Olga Natkovich
            Reporter:
            Olga Natkovich
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development