Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3888

Direct fetch doesn't differentiate between frontend and backend sides

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      While testing PIG-3558, I found that fetch mode doesn't set the task id in the jobconf before running. As a consequence, UdfContext#isFrontend() will always return true. Loaders and Storers which initialize themselves based on this flag may fail.

      1. PIG-3888.patch
        4 kB
        Lorand Bendig

        Activity

        Hide
        lbendig Lorand Bendig added a comment -

        The patch sets the task attempt id and also restores the jobconf at the end of the fetch task. Thus, the subsequent operators won't be affected by the messed up jobConf which could otherwise cause wrong initialization issues.

        Show
        lbendig Lorand Bendig added a comment - The patch sets the task attempt id and also restores the jobconf at the end of the fetch task. Thus, the subsequent operators won't be affected by the messed up jobConf which could otherwise cause wrong initialization issues.
        Hide
        cheolsoo Cheolsoo Park added a comment -

        +1.

        Show
        cheolsoo Cheolsoo Park added a comment - +1.
        Hide
        daijy Daniel Dai added a comment -

        Verified TestOrcStorage works with "opt.fetch" on with the patch. Patch committed to trunk. Thanks Lorand, Cheolsoo!

        Show
        daijy Daniel Dai added a comment - Verified TestOrcStorage works with "opt.fetch" on with the patch. Patch committed to trunk. Thanks Lorand, Cheolsoo!
        Hide
        lbendig Lorand Bendig added a comment -

        Daniel Dai, thanks for committing it. I've also checked it against all unit tests.

        Show
        lbendig Lorand Bendig added a comment - Daniel Dai , thanks for committing it. I've also checked it against all unit tests.

          People

          • Assignee:
            lbendig Lorand Bendig
            Reporter:
            lbendig Lorand Bendig
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development