Pig
  1. Pig
  2. PIG-879

Pig should provide a way for input location string in load statement to be passed as-is to the Loader

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.3.0
    • Fix Version/s: 0.7.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Due to multiquery optimization, Pig always converts the filenames to absolute URIs (see http://wiki.apache.org/pig/PigMultiQueryPerformanceSpecification - section about Incompatible Changes - Path Names and Schemes). This is necessary since the script may have "cd .." statements between load or store statements and if the load statements have relative paths, we would need to convert to absolute paths to know where to load/store from. To do this QueryParser.massageFilename() has the code below[1] which basically gives the fully qualified hdfs path

      However the issue with this approach is that if the filename string is something like "hdfs://localhost.localdomain:39125/user/bla/1,hdfs://localhost.localdomain:39125/user/bla/2", the code below[1] actually translates this to hdfs://localhost.localdomain:38264/user/bla/1,hdfs://localhost.localdomain:38264/user/bla/2 and throws an exception that it is an incorrect path.

      Some loaders may want to interpret the filenames (the input location string in the load statement) in any way they wish and may want Pig to not make absolute paths out of them.

      There are a few options to address this:
      1) A command line switch to indicate to Pig that pathnames in the script are all absolute and hence Pig should not alter them and pass them as-is to Loaders and Storers.
      2) A keyword in the load and store statements to indicate the same intent to pig
      3) A property which users can supply on cmdline or in pig.properties to indicate the same intent.
      4) A method in LoadFunc - relativeToAbsolutePath(String filename, String curDir) which does the conversion to absolute - this way Loader can chose to implement it as a noop.

      Thoughts?

      1. PIG-879.patch
        99 kB
        Richard Ding
      2. PIG-879.patch
        77 kB
        Richard Ding
      3. PIG-879.patch
        77 kB
        Richard Ding
      4. PIG-879.patch
        76 kB
        Richard Ding
      5. PIG-879.patch
        60 kB
        Richard Ding

        Issue Links

          Activity

          Hide
          Hong Tang added a comment -

          1) and 3) are kind of equivalent to user, and are preferred for customized loaders that do not wish pig to do the escaping at all.

          Show
          Hong Tang added a comment - 1) and 3) are kind of equivalent to user, and are preferred for customized loaders that do not wish pig to do the escaping at all.
          Hide
          Dmitriy V. Ryaboy added a comment -

          Having this be a global flag through properties wouldn't work for scripts that require both behaviors in different load statements.

          Maybe a boolean "performPathConversion" flag which is true by default, and can be overridden via the load statement?
          Custom Loaders could change what their default is.
          I think a boolean flag is more straightforward than a method you have to override with a no-op.

          Show
          Dmitriy V. Ryaboy added a comment - Having this be a global flag through properties wouldn't work for scripts that require both behaviors in different load statements. Maybe a boolean "performPathConversion" flag which is true by default, and can be overridden via the load statement? Custom Loaders could change what their default is. I think a boolean flag is more straightforward than a method you have to override with a no-op.
          Hide
          Thejas M Nair added a comment -

          The problem with 1 & 3 is that the setting is universal to the grunt shell or script.
          In cases where user wants to read from read from multiple sources with different loaders, it will be inconvenient to be forced to use absolute uri's for all of them.

          Show
          Thejas M Nair added a comment - The problem with 1 & 3 is that the setting is universal to the grunt shell or script. In cases where user wants to read from read from multiple sources with different loaders, it will be inconvenient to be forced to use absolute uri's for all of them.
          Hide
          Hong Tang added a comment -

          Both are valid arguments. The problem of 2) and 4) are that they require change to the load statement syntax or load-func api and would take longer to get there.

          I guess we could structure the fix in two phases: Phase One: supporting 1) and 3), so that we can have the minimum to move along without having to disable multi-query optimization completely. User should be able to modify the script to change all relative paths to absolute ones (the chance of such usage should be rare that most people should not be impacted). Phase Two: support either 2) or 4) (but I do not think we need both). And personally I think 4) would be better because loader should be the one that interprets the location string syntax.

          Show
          Hong Tang added a comment - Both are valid arguments. The problem of 2) and 4) are that they require change to the load statement syntax or load-func api and would take longer to get there. I guess we could structure the fix in two phases: Phase One: supporting 1) and 3), so that we can have the minimum to move along without having to disable multi-query optimization completely. User should be able to modify the script to change all relative paths to absolute ones (the chance of such usage should be rare that most people should not be impacted). Phase Two: support either 2) or 4) (but I do not think we need both). And personally I think 4) would be better because loader should be the one that interprets the location string syntax.
          Hide
          Milind Bhandarkar added a comment -

          I see some long term issues with all the approaches/options.

          First, not all loaders require a path. (e.g. DBLoader) Some paths (e.g. hftp:// or hsftp://) do not have a notion of relative or absolute. Indeed, the right way to fix this is to change the syntax of load and store statements, so that the loader itself deals with the path handling, and not pig. Second, take out copyToLocal, cp, mv, and all the dfs shell functionality from pig. These are side effects and impose a barrier for optimization. In the current form, they do not belong in a dataflow language. Grunt could still support it.

          Show
          Milind Bhandarkar added a comment - I see some long term issues with all the approaches/options. First, not all loaders require a path. (e.g. DBLoader) Some paths (e.g. hftp:// or hsftp://) do not have a notion of relative or absolute. Indeed, the right way to fix this is to change the syntax of load and store statements, so that the loader itself deals with the path handling, and not pig. Second, take out copyToLocal, cp, mv, and all the dfs shell functionality from pig. These are side effects and impose a barrier for optimization. In the current form, they do not belong in a dataflow language. Grunt could still support it.
          Hide
          Dmitriy V. Ryaboy added a comment -

          Milind – as far as hftp:// and hsftp:// are concerned, the multiquery spec says "Arguments used in a load statement that have a scheme other than "hdfs" or "file" will not be expanded and passed to the LoadFunc/Slicer unchanged. "

          Show
          Dmitriy V. Ryaboy added a comment - Milind – as far as hftp:// and hsftp:// are concerned, the multiquery spec says "Arguments used in a load statement that have a scheme other than "hdfs" or "file" will not be expanded and passed to the LoadFunc/Slicer unchanged. "
          Hide
          Pradeep Kamath added a comment -

          We may want to address this as part of implementing http://wiki.apache.org/pig/LoadStoreRedesignProposal. Option 4 seems the most extensible since loadfuncs get flexibility of dealing with the location string. At the same time we could provide some utility method to LoadFunc writers which they can use or refer to and this can be provided by pig as a static utility method (this can be the implementation that Pig internally uses in its builtin loaders)

          Show
          Pradeep Kamath added a comment - We may want to address this as part of implementing http://wiki.apache.org/pig/LoadStoreRedesignProposal . Option 4 seems the most extensible since loadfuncs get flexibility of dealing with the location string. At the same time we could provide some utility method to LoadFunc writers which they can use or refer to and this can be provided by pig as a static utility method (this can be the implementation that Pig internally uses in its builtin loaders)
          Hide
          Richard Ding added a comment -

          As a related issue, there is a feature in Pig right now that allows user to specify a local file in a load statement even if Pig is running in MapReduce mode. Namely,

          A = load 'file:test/org/apache/pig/test/data/passwd' using PigStorage(':');
          

          is a valid Pig statement. Internally Pig moves the file from the local file system to the HDFS file system and gives the corresponding HDFS URI to the loader:

          hdfs://localhost.localdomain:37575/tmp/temp957104276/tmp124591329
          

          As we move to the new Load/Store API, there are two options:

          1. Stop supporting this feature. The above load statement can be replaced by the following statements:

          copyFromLocal ./test/org/apache/pig/test/data/passw ./passw
          A = load './passw' using PigStorage(':');
          

          2. The default implementation of relativeToAbsolutePath(String location, Path curDir) method will move the local file (specified by location) to the HDFS and return the corresponding HDFS URI.

          Any comments on these options? Especially does anyone have strong opinion against choosing option 1?

          Show
          Richard Ding added a comment - As a related issue, there is a feature in Pig right now that allows user to specify a local file in a load statement even if Pig is running in MapReduce mode. Namely, A = load 'file:test/org/apache/pig/test/data/passwd' using PigStorage(':'); is a valid Pig statement. Internally Pig moves the file from the local file system to the HDFS file system and gives the corresponding HDFS URI to the loader: hdfs: //localhost.localdomain:37575/tmp/temp957104276/tmp124591329 As we move to the new Load/Store API, there are two options: 1. Stop supporting this feature. The above load statement can be replaced by the following statements: copyFromLocal ./test/org/apache/pig/test/data/passw ./passw A = load './passw' using PigStorage(':'); 2. The default implementation of relativeToAbsolutePath(String location, Path curDir) method will move the local file (specified by location) to the HDFS and return the corresponding HDFS URI. Any comments on these options? Especially does anyone have strong opinion against choosing option 1?
          Hide
          Richard Ding added a comment -

          This patch implements the option 1 of the above comment and is for load-store-redesign branch.

          Show
          Richard Ding added a comment - This patch implements the option 1 of the above comment and is for load-store-redesign branch.
          Hide
          Richard Ding added a comment -

          The test-patch run results:

          [exec] -1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 23 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] -1 release audit. The applied patch generated 333 release audit warnings (more than the trunk's current 331 warnings).

          Show
          Richard Ding added a comment - The test-patch run results: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 23 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 333 release audit warnings (more than the trunk's current 331 warnings).
          Hide
          Richard Ding added a comment -

          This patch added apache header to the new unit test file. The remaining audit warning is html related.

          Show
          Richard Ding added a comment - This patch added apache header to the new unit test file. The remaining audit warning is html related.
          Hide
          Richard Ding added a comment -

          This patch is generated after applying the patch for PIG-1094. Here is the test-patch results:

          [exec] -1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 41 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] -1 release audit. The applied patch generated 332 release audit warnings (more than the trunk's current 331 warnings).

          The only additional audit warning is about html:

          < [java] !????? /homes/rding/apache-pig/load-store-redesign/build/pig-0.6.0-dev/docs/jdiff/changes/org.apache.pig.ReversibleLoadStoreFunc.html

          Show
          Richard Ding added a comment - This patch is generated after applying the patch for PIG-1094 . Here is the test-patch results: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 41 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 332 release audit warnings (more than the trunk's current 331 warnings). The only additional audit warning is about html: < [java] !????? /homes/rding/apache-pig/load-store-redesign/build/pig-0.6.0-dev/docs/jdiff/changes/org.apache.pig.ReversibleLoadStoreFunc.html
          Hide
          Pradeep Kamath added a comment -

          In TestLoad.java it looks like a couple of tests have been commented since they should run in local mode.
          Since TestLoad is currently working in MAPREDUCE mode and we no longer support file:<path> in MAPREDUCE mode, we
          should probably change TestLoad to run each test in both local and map-red mode so we test relative to absolute
          path conversion in both modes.

          I think there is a jira opened against hadoop to make the method to parse file globs public - currently
          the same code is in the patch - we should put a comment in the code that we should use Hadoop's method when
          the jira-XXXX is fixed

          While creating a LOStore in the QueryParser, we should be calling StoreFunc's relToAbsoluteStoreLocation() method.
          We should add a comment in StoreFunc interface that implementations can just use LoadFunc's static methid to implement
          this method.

          Show
          Pradeep Kamath added a comment - In TestLoad.java it looks like a couple of tests have been commented since they should run in local mode. Since TestLoad is currently working in MAPREDUCE mode and we no longer support file:<path> in MAPREDUCE mode, we should probably change TestLoad to run each test in both local and map-red mode so we test relative to absolute path conversion in both modes. I think there is a jira opened against hadoop to make the method to parse file globs public - currently the same code is in the patch - we should put a comment in the code that we should use Hadoop's method when the jira-XXXX is fixed While creating a LOStore in the QueryParser, we should be calling StoreFunc's relToAbsoluteStoreLocation() method. We should add a comment in StoreFunc interface that implementations can just use LoadFunc's static methid to implement this method.
          Hide
          Richard Ding added a comment -

          This patch addressed the above comments.

          [exec] -1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 53 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] -1 release audit. The applied patch generated 339 release audit warnings (more than the trunk's current 338 warnings).

          Show
          Richard Ding added a comment - This patch addressed the above comments. [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 53 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] -1 release audit. The applied patch generated 339 release audit warnings (more than the trunk's current 338 warnings).
          Hide
          Pradeep Kamath added a comment -

          +1, patch committed on load-store-redesign branch with minor change in TestLoad to correctly set up file on MiniCluster.

          Show
          Pradeep Kamath added a comment - +1, patch committed on load-store-redesign branch with minor change in TestLoad to correctly set up file on MiniCluster.

            People

            • Assignee:
              Richard Ding
              Reporter:
              Pradeep Kamath
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development