Pig
  1. Pig
  2. PIG-3223

AvroStorage does not handle comma separated input paths

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.10.0, 0.11
    • Fix Version/s: 0.12.0, 0.11.2
    • Component/s: piggybank
    • Labels:
      None

      Description

      In pig 0.11, a patch was issued to AvroStorage to support globs and comma separated input paths (PIG-2492). While this function works fine for glob-formatted input paths, it fails when issued a standard comma separated list of paths. fs.globStatus does not seem to be able to parse out such a list, and a java.net.URISyntaxException is thrown when toURI is called on the path.

      I have a working fix for this, but it's extremely ugly (basically checking if the string of input paths is globbed, otherwise splitting on ","). I'm sure there's a more elegant solution. I'd be happy to post the relevant methods and "fixes" if necessary.

      1. AvroStorage.patch
        28 kB
        Michael Kramer
      2. AvroStorage.patch-2
        28 kB
        Michael Kramer
      3. AvroStorageUtils.patch
        26 kB
        Michael Kramer
      4. AvroStorageUtils.patch-2
        26 kB
        Michael Kramer
      5. PIG-3223.branch-0.11.patch.txt
        8 kB
        Johnny Zhang
      6. PIG-3223.patch.txt
        8 kB
        Johnny Zhang
      7. PIG-3223.patch.txt
        8 kB
        Johnny Zhang
      8. PIG-3223.patch.txt
        8 kB
        Johnny Zhang
      9. PIG-3223.patch.txt
        8 kB
        Johnny Zhang
      10. PIG-3223.patch.txt
        2 kB
        Johnny Zhang

        Issue Links

          Activity

            People

            • Assignee:
              Johnny Zhang
              Reporter:
              Michael Kramer
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development