Uploaded image for project: 'Pig'
  1. Pig
  2. PIG-3446 Umbrella jira for Pig on Tez
  3. PIG-4039

New interface for resetting static variables for jvm reuse

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: tez
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In Tez, when there is container reuse, static variables, thread locals, etc have to be reinitialized to avoid memory leak or having wrong values. For the short term, we ended up making some of the static variables public or adding destroy method to each class which is hacky. Also it does not help users who will want something similar to be done in their UDFs or LoadFunc. Need to define a interface with a reset/destroy method and find all loaded classes implementing that interface and call destroy on them in PigProcessor.close(). ServiceLoader and annotations are some of the ways to find classes implementing an interface and there are other libraries as well. Need to find the best and fastest way to do that.

        Attachments

        1. PIG-4039-1.patch
          20 kB
          Rohini Palaniswamy
        2. PIG-4039-2.patch
          20 kB
          Rohini Palaniswamy

          Activity

            People

            • Assignee:
              rohini Rohini Palaniswamy
              Reporter:
              rohini Rohini Palaniswamy
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: