Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: tez-branch
    • Component/s: tez
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      PigCombiner.sJobContext and PhysicalOperator.reporter hold references to the WrappedReducer$Context which in turn holds TezOutputContextImpl which holds references to the buffers in DefaultSorter. This was causing OOM after the container was reused 2 or 3 times. Debugged this with L17.pig in pigmix.

      1. PIG-3797-1.patch
        9 kB
        Rohini Palaniswamy

        Activity

        Hide
        Rohini Palaniswamy added a comment -

        Committed to Tez branch. Thanks Cheolsoo for the review.

        Show
        Rohini Palaniswamy added a comment - Committed to Tez branch. Thanks Cheolsoo for the review.
        Hide
        Cheolsoo Park added a comment -

        +1. Make sense. Thank you for the clarification!

        Show
        Cheolsoo Park added a comment - +1. Make sense. Thank you for the clarification!
        Hide
        Rohini Palaniswamy added a comment -

        ThreadLocal.remove() only "Removes the current thread's value for this thread-local variable". Reinitializing it so that the old object gets garbage collected. I did not find any equivalent method to clear thread-local variable of all threads. Since the PigProcessor.close() can be called by Tez using any thread, just removing the current thread's value will not work.

        Show
        Rohini Palaniswamy added a comment - ThreadLocal.remove() only "Removes the current thread's value for this thread-local variable". Reinitializing it so that the old object gets garbage collected. I did not find any equivalent method to clear thread-local variable of all threads. Since the PigProcessor.close() can be called by Tez using any thread, just removing the current thread's value will not work.
        Hide
        Cheolsoo Park added a comment -

        Why not use ThreadLocal.remove() to reinitialize the ThreadLocal variables?

        +        // Reset static variables cleared for avoiding OOM.
        +        // TODO: Figure out a cleaner way to do this. ThreadLocals actually can be avoided all together
        +        // for mapreduce/tez mode and just used for Local mode.
        +        PhysicalOperator.reporter = new ThreadLocal<PigProgressable>();
        +        PigMapReduce.sJobConfInternal = new ThreadLocal<Configuration>();
        
        +        // Avoid memory leak. ThreadLocals especially leak a lot of memory.
        +        PhysicalOperator.reporter = new ThreadLocal<PigProgressable>();
        +        PigMapReduce.sJobConfInternal = new ThreadLocal<Configuration>();
        

        Otherwise, looks good to me.

        Show
        Cheolsoo Park added a comment - Why not use ThreadLocal.remove() to reinitialize the ThreadLocal variables? + // Reset static variables cleared for avoiding OOM. + // TODO: Figure out a cleaner way to do this . ThreadLocals actually can be avoided all together + // for mapreduce/tez mode and just used for Local mode. + PhysicalOperator.reporter = new ThreadLocal<PigProgressable>(); + PigMapReduce.sJobConfInternal = new ThreadLocal<Configuration>(); + // Avoid memory leak. ThreadLocals especially leak a lot of memory. + PhysicalOperator.reporter = new ThreadLocal<PigProgressable>(); + PigMapReduce.sJobConfInternal = new ThreadLocal<Configuration>(); Otherwise, looks good to me.

          People

          • Assignee:
            Rohini Palaniswamy
            Reporter:
            Rohini Palaniswamy
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development