Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-2553

Pig shouldn't allow attempts to write multiple relations into same directory

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available
    • Release Note:
      To enforce strict checking of output location, set "pig.location.check.strict=true". By default, it is false.

      Description

      We've seen multiple occasions where users accidentally try to store 2 or more different relations to the same destination directory. Currently, this passes the Pig planner and fails on MR side due to concurrent attempts to create the same part file on the reducer. This is extremely confusing to the user, and hard to debug.
      We should instead fail their scripts before they are even submitted, since we can identify the erroneous condition from the beginning.

        Attachments

        1. PIG-2553.patch
          2 kB
          Prashant Kommireddi
        2. PIG-2553_1.patch
          8 kB
          Prashant Kommireddi
        3. PIG-2553_2.patch
          9 kB
          Prashant Kommireddi

          Activity

            People

            • Assignee:
              prkommireddi Prashant Kommireddi
              Reporter:
              dvryaboy Dmitriy V. Ryaboy
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: