Pig
  1. Pig
  2. PIG-3888

Direct fetch doesn't differentiate between frontend and backend sides

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      While testing PIG-3558, I found that fetch mode doesn't set the task id in the jobconf before running. As a consequence, UdfContext#isFrontend() will always return true. Loaders and Storers which initialize themselves based on this flag may fail.

      1. PIG-3888.patch
        4 kB
        Lorand Bendig

        Activity

        Hide
        Lorand Bendig added a comment -

        The patch sets the task attempt id and also restores the jobconf at the end of the fetch task. Thus, the subsequent operators won't be affected by the messed up jobConf which could otherwise cause wrong initialization issues.

        Show
        Lorand Bendig added a comment - The patch sets the task attempt id and also restores the jobconf at the end of the fetch task. Thus, the subsequent operators won't be affected by the messed up jobConf which could otherwise cause wrong initialization issues.
        Hide
        Cheolsoo Park added a comment -

        +1.

        Show
        Cheolsoo Park added a comment - +1.
        Hide
        Daniel Dai added a comment -

        Verified TestOrcStorage works with "opt.fetch" on with the patch. Patch committed to trunk. Thanks Lorand, Cheolsoo!

        Show
        Daniel Dai added a comment - Verified TestOrcStorage works with "opt.fetch" on with the patch. Patch committed to trunk. Thanks Lorand, Cheolsoo!
        Hide
        Lorand Bendig added a comment -

        Daniel Dai, thanks for committing it. I've also checked it against all unit tests.

        Show
        Lorand Bendig added a comment - Daniel Dai , thanks for committing it. I've also checked it against all unit tests.

          People

          • Assignee:
            Lorand Bendig
            Reporter:
            Lorand Bendig
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development