Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3223

AvroStorage does not handle comma separated input paths

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0, 0.11
    • Fix Version/s: 0.12.0, 0.11.2
    • Component/s: piggybank
    • Labels:
      None

      Description

      In pig 0.11, a patch was issued to AvroStorage to support globs and comma separated input paths (PIG-2492). While this function works fine for glob-formatted input paths, it fails when issued a standard comma separated list of paths. fs.globStatus does not seem to be able to parse out such a list, and a java.net.URISyntaxException is thrown when toURI is called on the path.

      I have a working fix for this, but it's extremely ugly (basically checking if the string of input paths is globbed, otherwise splitting on ","). I'm sure there's a more elegant solution. I'd be happy to post the relevant methods and "fixes" if necessary.

        Attachments

        1. AvroStorage.patch
          28 kB
          Michael Kramer
        2. AvroStorage.patch-2
          28 kB
          Michael Kramer
        3. AvroStorageUtils.patch
          26 kB
          Michael Kramer
        4. AvroStorageUtils.patch-2
          26 kB
          Michael Kramer
        5. PIG-3223.branch-0.11.patch.txt
          8 kB
          Johnny Zhang
        6. PIG-3223.patch.txt
          8 kB
          Johnny Zhang
        7. PIG-3223.patch.txt
          8 kB
          Johnny Zhang
        8. PIG-3223.patch.txt
          8 kB
          Johnny Zhang
        9. PIG-3223.patch.txt
          8 kB
          Johnny Zhang
        10. PIG-3223.patch.txt
          2 kB
          Johnny Zhang

          Issue Links

            Activity

              People

              • Assignee:
                dreambird Johnny Zhang
                Reporter:
                mkramer Michael Kramer
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: