Pig
  1. Pig
  2. PIG-3978

Container reuse does not across PigServer

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: tez
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Container reuse across PigServer does not happen in current code. We use different TezResourceManager per PigServer, and different TezResourceManager will upload LocalResource to different staging directory, so LocalResource is different for different PigServer, thus no container reuse across PigServer. This will affect Pig embedding where multiple PigServer is used for a Pig script.

      The issue is partly due to PIG-3785 which we switch TezResourceManager to non-singleton. The root cause for PIG-3785 is not the singleton issue. The real issue is TezResourceManager.addTezResource is not synchronized, there is a race condition when one PigServer already upload the resource and use it in container, and the other PigServer overwrite at the same time.

        Activity

        Hide
        Daniel Dai added a comment -
        Show
        Daniel Dai added a comment - RB link: https://reviews.apache.org/r/22155/
        Hide
        Daniel Dai added a comment -

        Patch committed to trunk. Thanks for Rohini for review (review notes is on RB)!

        Show
        Daniel Dai added a comment - Patch committed to trunk. Thanks for Rohini for review (review notes is on RB)!

          People

          • Assignee:
            Daniel Dai
            Reporter:
            Daniel Dai
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development