Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Duplicate
-
None
-
None
-
None
-
None
-
Incompatible change
Description
As part of the multiquery optimization work there's a need to use absolute paths for load and store operations (because the current directory changes during the execution of the script). In order to do so, the suggestion is to change the semantics of the location/filename string used in LoadFunc and Slicer/Slice.
The proposed change is:
- Load locations without a scheme part are expected to be hdfs (mapreduce mode) or local (local mode) paths
- Any hdfs or local path will be translated to a fully qualified absolute path before it is handed to either a LoadFunc or Slicer
- Any scheme other than file or hdfs will result in the load path be passed through to the LoadFunc or Slicer without any modification.
Example:
If you have a LoadFunc that reads from a database, right now the following could be used:
{{{
a = load 'table' using DBLoader();
}}}
With the proposed changes table would be translated into an hdfs path though ("hdfs://..../table"). Probably not what the loader wants to see. So in order to make this work one would use:
{{{
a = load 'sql://table' using DBLoader();
}}}
Now the DBLoader would see the unchanged string "sql://table". And pig will not use the string as an hdfs location.
This is an incompatible change but it's hopefully few existing Slicers/Loaders that are affected. This behavior is part of the multiquery work and can be turned off (reverted back) by using the "no_multiquery" flag.