Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-1162

Tez leaks CodecPool buffers

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.5.0
    • 0.5.0, 0.4.1
    • None
    • None

    Description

      Tez Fetcher currently leaks a codec pool (~32kb) for each partition fetched

      This causes task failures due to the direct buffers allocated by the codec, which are not freed until a full GC (but whose allocation does not trigger GC).

      It is possible to perform the entire shuffle operations without ever triggering a full GC, the following case

      Container exited with a non-zero exit code 143
      ], AttemptID:attempt_1399351577718_2330_1_05_000020_3 Info:Container container_1399351577718_2330_01_000310 COMPLETED with diagnostics set to [Container [pid=1734,containerID=container_1399351577718_2330_01_000310] is running beyond physical memory limits. Current usage: 4.1 GB of 4 GB physical memory used; 5.4 GB of 40 GB virtual memory used. Killing container.
      
      container_1399351577718_2330_01_000365/ $ grep -ri "CodecPool.*brand-new" syslog* | wc -l
      
      6988
      

      That is approx ~436Mb leak on a JVM spun up with -Xmx3500m & 4096m container.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            gopalv Gopal Vijayaraghavan
            gopalv Gopal Vijayaraghavan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment