Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
-
None
Description
I have a list of paths stored in a filenames.txt. I would like to load them all using a single LOAD command. The paths don't conform to one or more regexes, so they have to specified individually.
So far I've used the -param option with pig to specify them. But it results in an extremely long commandline and I'm afraid I wont be able to scale my script.
shell : pig -param read_paths=<my-long-list-of-paths> something.pig
something.pig : requests = LOAD '$read_paths' USING PigStorage(',');