[PIG-758] Converting load/store locations into fully qualified absolute paths

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.7.0
• Hadoop Flags: Incompatible change
• Assignee: Unassigned
• Reporter: Gunther Hagleitner
• Created: 08/Apr/09 19:56
• Updated: 14/May/10 06:45
• Resolved: 28/Jan/10 00:14

Description

As part of the multiquery optimization work there is a need to use absolute paths for load and store operations (because the current directory changes during the execution of the script). In order to do so, we are suggesting a change to the semantics of the location/filename string used in LoadFunc and Slicer/Slice.

The proposed change is:

• Load locations without a scheme part are expected to be hdfs (mapreduce mode) or local (local mode) paths
• Any hdfs or local path will be translated to a fully qualified absolute path before it is handed to either a LoadFunc or Slicer
• Any scheme other than "file" or "hdfs" will result in the load path to be passed through to the LoadFunc or Slicer without any modification.

Example:

If you have a LoadFunc that reads from a database, in the current system the following could be used:

a = load 'table' using DBLoader();

With the proposed changes table would be translated into an hdfs path though ("hdfs://..../table"). Probably not what the DBLoader would want to see. In order to make it work one could use:

a = load 'sql://table' using DBLoader();

Now the DBLoader would see the unchanged string "sql://table".

This is an incompatible change, but hopefully not affecting many existing Loaders/Slicers. Since this is needed with the multiquery feature, the behavior can be reverted back by using the "no_multiquery" pig flag.