Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3400

FS commands do not work with S3 paths

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: grunt
    • Labels:
      None

      Description

      To reproduce issues, run the following commands w/ S3 paths:

      pig -e 'ls s3://<path>'
      pig -e 'mkdir s3://<path>'
      pig -e 'cp s3://<path1> s3://<path2>'
      pig -e 'mv s3://<path1> s3://<path2>'
      pig -e 'copyToLocal s3://<path>/<file> .'
      pig -e 'copyFromLocal <file> s3://<path>'
      

      As of now, none of these commands works in Apache Pig, whereas they do in EMR Pig. The problem is that in GruntParser, DataStorage is constructed using the default file system provided by configuration, and s3 paths are not recognized.

      Instead, we should construct DataStorage based on the given URL. For example,

      DataStorage dfs = new HDataStorage(new Path(path).toUri(), mConf);
      

        Attachments

        1. PIG-3400-3.patch
          31 kB
          Cheolsoo Park
        2. PIG-3400-2.patch
          10 kB
          Cheolsoo Park
        3. PIG-3400.patch
          10 kB
          Cheolsoo Park

          Activity

            People

            • Assignee:
              cheolsoo Cheolsoo Park
              Reporter:
              cheolsoo Cheolsoo Park
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: