Pig
  1. Pig
  2. PIG-2553

Pig shouldn't allow attempts to write multiple relations into same directory

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available
    • Release Note:
      To enforce strict checking of output location, set "pig.location.check.strict=true". By default, it is false.

      Description

      We've seen multiple occasions where users accidentally try to store 2 or more different relations to the same destination directory. Currently, this passes the Pig planner and fails on MR side due to concurrent attempts to create the same part file on the reducer. This is extremely confusing to the user, and hard to debug.
      We should instead fail their scripts before they are even submitted, since we can identify the erroneous condition from the beginning.

      1. PIG-2553_1.patch
        8 kB
        Prashant Kommireddi
      2. PIG-2553_2.patch
        9 kB
        Prashant Kommireddi
      3. PIG-2553.patch
        2 kB
        Prashant Kommireddi

        Activity

        Dmitriy V. Ryaboy created issue -
        Prashant Kommireddi made changes -
        Field Original Value New Value
        Attachment PIG-2553.patch [ 12551946 ]
        Prashant Kommireddi made changes -
        Assignee Prashant Kommireddi [ prkommireddi ]
        Prashant Kommireddi made changes -
        Attachment PIG-2553_1.patch [ 12554307 ]
        Prashant Kommireddi made changes -
        Attachment PIG-2553_2.patch [ 12568542 ]
        Prashant Kommireddi made changes -
        Patch Info Patch Available [ 10042 ]
        Cheolsoo Park made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Release Note To enforce strict checking of output location, set "pig.location.check.strict=true". By default, it is false.
        Fix Version/s 0.12 [ 12323380 ]
        Resolution Fixed [ 1 ]
        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Prashant Kommireddi
            Reporter:
            Dmitriy V. Ryaboy
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development