Pig
  1. Pig
  2. PIG-3978

Container reuse does not across PigServer

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: tez
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Container reuse across PigServer does not happen in current code. We use different TezResourceManager per PigServer, and different TezResourceManager will upload LocalResource to different staging directory, so LocalResource is different for different PigServer, thus no container reuse across PigServer. This will affect Pig embedding where multiple PigServer is used for a Pig script.

      The issue is partly due to PIG-3785 which we switch TezResourceManager to non-singleton. The root cause for PIG-3785 is not the singleton issue. The real issue is TezResourceManager.addTezResource is not synchronized, there is a race condition when one PigServer already upload the resource and use it in container, and the other PigServer overwrite at the same time.

        Activity

        Daniel Dai created issue -
        Daniel Dai made changes -
        Field Original Value New Value
        Attachment PIG-3978-1.patch [ 12647784 ]
        Hide
        Daniel Dai added a comment -
        Show
        Daniel Dai added a comment - RB link: https://reviews.apache.org/r/22155/
        Hide
        Daniel Dai added a comment -

        Patch committed to trunk. Thanks for Rohini for review (review notes is on RB)!

        Show
        Daniel Dai added a comment - Patch committed to trunk. Thanks for Rohini for review (review notes is on RB)!
        Daniel Dai made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Resolution Fixed [ 1 ]
        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        2d 55m 1 Daniel Dai 02/Jun/14 22:13
        Resolved Resolved Closed Closed
        171d 8h 45m 1 Daniel Dai 21/Nov/14 05:59

          People

          • Assignee:
            Daniel Dai
            Reporter:
            Daniel Dai
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development