Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3978

Container reuse does not across PigServer

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 0.14.0
    • tez
    • None
    • Reviewed

    Description

      Container reuse across PigServer does not happen in current code. We use different TezResourceManager per PigServer, and different TezResourceManager will upload LocalResource to different staging directory, so LocalResource is different for different PigServer, thus no container reuse across PigServer. This will affect Pig embedding where multiple PigServer is used for a Pig script.

      The issue is partly due to PIG-3785 which we switch TezResourceManager to non-singleton. The root cause for PIG-3785 is not the singleton issue. The real issue is TezResourceManager.addTezResource is not synchronized, there is a race condition when one PigServer already upload the resource and use it in container, and the other PigServer overwrite at the same time.

      Attachments

        1. PIG-3978-1.patch
          16 kB
          Daniel Dai

        Activity

          People

            daijy Daniel Dai
            daijy Daniel Dai
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: