Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-4703

TezOperator.stores shall not ship to backend

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.16.0, 0.15.1
    • Component/s: tez
    • Labels:
      None

      Description

      We end up of shipping all physical plan to the backend. One direct result is UDFContext is not set in OutputCommitter. Here is how:
      1. PigGraceShuffleVertexManager deserialize "pig.tez.plan". TezOperator.stores reference physical plan so physical plan get deserialize.
      2. When deserialize POUserFunc, POUserFunc.instantiateFunc is invoked. Inside it, POUserFunc.setFuncInputSchema would use UDFContext and leave an entry there (though it is empty map).
      3. MapRedUtil.setupUDFContext would only deserialize if UDFContext is empty, since we already touch UDFContext with some maps, UDFContext is not deserialized

        Attachments

        1. PIG-4703-1.patch
          11 kB
          Daniel Dai
        2. PIG-4703-2.patch
          14 kB
          Daniel Dai

          Issue Links

            Activity

              People

              • Assignee:
                daijy Daniel Dai
                Reporter:
                daijy Daniel Dai
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: