Pig
  1. Pig
  2. PIG-3548

Allow pig to load multiple paths specified in a filenames.txt

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I have a list of paths stored in a filenames.txt. I would like to load them all using a single LOAD command. The paths don't conform to one or more regexes, so they have to specified individually.

      So far I've used the -param option with pig to specify them. But it results in an extremely long commandline and I'm afraid I wont be able to scale my script.

      shell : pig -param read_paths=<my-long-list-of-paths> something.pig
      something.pig : requests = LOAD '$read_paths' USING PigStorage(',');

        Activity

        Hide
        Alan Gates added a comment -

        Could you store the parameters in a file rather than specify them on the command line? See http://pig.apache.org/docs/r0.12.0/cont.html#Parameter-Sub for details.

        Show
        Alan Gates added a comment - Could you store the parameters in a file rather than specify them on the command line? See http://pig.apache.org/docs/r0.12.0/cont.html#Parameter-Sub for details.

          People

          • Assignee:
            Unassigned
            Reporter:
            Madhavi Nadig
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:

              Development