Uploaded image for project: 'Apache Tez'
  1. Apache Tez
  2. TEZ-1162

Tez leaks CodecPool buffers

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.5.0
    • Fix Version/s: 0.5.0, 0.4.1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:

      Description

      Tez Fetcher currently leaks a codec pool (~32kb) for each partition fetched

      This causes task failures due to the direct buffers allocated by the codec, which are not freed until a full GC (but whose allocation does not trigger GC).

      It is possible to perform the entire shuffle operations without ever triggering a full GC, the following case

      Container exited with a non-zero exit code 143
      ], AttemptID:attempt_1399351577718_2330_1_05_000020_3 Info:Container container_1399351577718_2330_01_000310 COMPLETED with diagnostics set to [Container [pid=1734,containerID=container_1399351577718_2330_01_000310] is running beyond physical memory limits. Current usage: 4.1 GB of 4 GB physical memory used; 5.4 GB of 40 GB virtual memory used. Killing container.
      
      container_1399351577718_2330_01_000365/ $ grep -ri "CodecPool.*brand-new" syslog* | wc -l
      
      6988
      

      That is approx ~436Mb leak on a JVM spun up with -Xmx3500m & 4096m container.

        Attachments

        1. TEZ-1162.1.patch
          1 kB
          Gopal Vijayaraghavan

          Issue Links

            Activity

              People

              • Assignee:
                gopalv Gopal Vijayaraghavan
                Reporter:
                gopalv Gopal Vijayaraghavan
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: