Pig
  1. Pig
  2. PIG-3888

Direct fetch doesn't differentiate between frontend and backend sides

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      While testing PIG-3558, I found that fetch mode doesn't set the task id in the jobconf before running. As a consequence, UdfContext#isFrontend() will always return true. Loaders and Storers which initialize themselves based on this flag may fail.

      1. PIG-3888.patch
        4 kB
        Lorand Bendig

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        10m 20s 1 Lorand Bendig 14/Apr/14 12:02
        Patch Available Patch Available Resolved Resolved
        8h 20m 1 Daniel Dai 14/Apr/14 20:22
        Resolved Resolved Closed Closed
        83d 22h 45m 1 Daniel Dai 07/Jul/14 19:07
        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Lorand Bendig added a comment -

        Daniel Dai, thanks for committing it. I've also checked it against all unit tests.

        Show
        Lorand Bendig added a comment - Daniel Dai , thanks for committing it. I've also checked it against all unit tests.
        Daniel Dai made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Fix Version/s 0.13.0 [ 12324971 ]
        Resolution Fixed [ 1 ]
        Hide
        Daniel Dai added a comment -

        Verified TestOrcStorage works with "opt.fetch" on with the patch. Patch committed to trunk. Thanks Lorand, Cheolsoo!

        Show
        Daniel Dai added a comment - Verified TestOrcStorage works with "opt.fetch" on with the patch. Patch committed to trunk. Thanks Lorand, Cheolsoo!
        Hide
        Cheolsoo Park added a comment -

        +1.

        Show
        Cheolsoo Park added a comment - +1.
        Lorand Bendig made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Lorand Bendig made changes -
        Field Original Value New Value
        Attachment PIG-3888.patch [ 12640049 ]
        Hide
        Lorand Bendig added a comment -

        The patch sets the task attempt id and also restores the jobconf at the end of the fetch task. Thus, the subsequent operators won't be affected by the messed up jobConf which could otherwise cause wrong initialization issues.

        Show
        Lorand Bendig added a comment - The patch sets the task attempt id and also restores the jobconf at the end of the fetch task. Thus, the subsequent operators won't be affected by the messed up jobConf which could otherwise cause wrong initialization issues.
        Lorand Bendig created issue -

          People

          • Assignee:
            Lorand Bendig
            Reporter:
            Lorand Bendig
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development