Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: tez-branch
    • Component/s: tez
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      PigCombiner.sJobContext and PhysicalOperator.reporter hold references to the WrappedReducer$Context which in turn holds TezOutputContextImpl which holds references to the buffers in DefaultSorter. This was causing OOM after the container was reused 2 or 3 times. Debugged this with L17.pig in pigmix.

      1. PIG-3797-1.patch
        9 kB
        Rohini Palaniswamy

        Activity

        Rohini Palaniswamy created issue -
        Rohini Palaniswamy made changes -
        Field Original Value New Value
        Component/s tez [ 12321016 ]
        Rohini Palaniswamy made changes -
        Attachment PIG-3797-1.patch [ 12633048 ]
        Rohini Palaniswamy made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Rohini Palaniswamy made changes -
        Attachment PIG-3797-1.patch [ 12633048 ]
        Rohini Palaniswamy made changes -
        Attachment PIG-3797-1.patch [ 12633090 ]
        Hide
        Cheolsoo Park added a comment -

        Why not use ThreadLocal.remove() to reinitialize the ThreadLocal variables?

        +        // Reset static variables cleared for avoiding OOM.
        +        // TODO: Figure out a cleaner way to do this. ThreadLocals actually can be avoided all together
        +        // for mapreduce/tez mode and just used for Local mode.
        +        PhysicalOperator.reporter = new ThreadLocal<PigProgressable>();
        +        PigMapReduce.sJobConfInternal = new ThreadLocal<Configuration>();
        
        +        // Avoid memory leak. ThreadLocals especially leak a lot of memory.
        +        PhysicalOperator.reporter = new ThreadLocal<PigProgressable>();
        +        PigMapReduce.sJobConfInternal = new ThreadLocal<Configuration>();
        

        Otherwise, looks good to me.

        Show
        Cheolsoo Park added a comment - Why not use ThreadLocal.remove() to reinitialize the ThreadLocal variables? + // Reset static variables cleared for avoiding OOM. + // TODO: Figure out a cleaner way to do this . ThreadLocals actually can be avoided all together + // for mapreduce/tez mode and just used for Local mode. + PhysicalOperator.reporter = new ThreadLocal<PigProgressable>(); + PigMapReduce.sJobConfInternal = new ThreadLocal<Configuration>(); + // Avoid memory leak. ThreadLocals especially leak a lot of memory. + PhysicalOperator.reporter = new ThreadLocal<PigProgressable>(); + PigMapReduce.sJobConfInternal = new ThreadLocal<Configuration>(); Otherwise, looks good to me.
        Hide
        Rohini Palaniswamy added a comment -

        ThreadLocal.remove() only "Removes the current thread's value for this thread-local variable". Reinitializing it so that the old object gets garbage collected. I did not find any equivalent method to clear thread-local variable of all threads. Since the PigProcessor.close() can be called by Tez using any thread, just removing the current thread's value will not work.

        Show
        Rohini Palaniswamy added a comment - ThreadLocal.remove() only "Removes the current thread's value for this thread-local variable". Reinitializing it so that the old object gets garbage collected. I did not find any equivalent method to clear thread-local variable of all threads. Since the PigProcessor.close() can be called by Tez using any thread, just removing the current thread's value will not work.
        Hide
        Cheolsoo Park added a comment -

        +1. Make sense. Thank you for the clarification!

        Show
        Cheolsoo Park added a comment - +1. Make sense. Thank you for the clarification!
        Hide
        Rohini Palaniswamy added a comment -

        Committed to Tez branch. Thanks Cheolsoo for the review.

        Show
        Rohini Palaniswamy added a comment - Committed to Tez branch. Thanks Cheolsoo for the review.
        Rohini Palaniswamy made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags Reviewed [ 10343 ]
        Resolution Fixed [ 1 ]
        Daniel Dai made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        2m 7s 1 Rohini Palaniswamy 06/Mar/14 05:45
        Patch Available Patch Available Resolved Resolved
        2d 2h 55m 1 Rohini Palaniswamy 08/Mar/14 08:41
        Resolved Resolved Closed Closed
        257d 21h 17m 1 Daniel Dai 21/Nov/14 05:59

          People

          • Assignee:
            Rohini Palaniswamy
            Reporter:
            Rohini Palaniswamy
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development