[TRAFODION-1265] LP Bug: 1463179 - VSBB update causes TM heap leak

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.1 (pre-incubation)
• Component/s: dtm
• Labels: heap leak vsbb
• External issue URL: https://bugs.launchpad.net/bugs/1463179
• Assignee: Joanie Cooper
• Reporter: Buddy Wilbanks
• Created: 22/Jul/15 18:20
• Updated: 26/Aug/15 20:33
• Resolved: 26/Aug/15 20:33

Description

Since the May 1 executor checkin that turned on VSBB update, the TM will leak memory to the point of an OOM condition after only 26 hours of the longevity test. Pretty sure it will be much quicker if we limit longevity to only perform the delivery transaction, which is the only VSBB invoker.

We've collected jmaps for some of the TMs on zircon4, and core files for analysis. Located on /home/squser4/fqw/TMdumps. You can see the corefile size increasing as well as the old space usage. Here's the first and last corefiles taken for pid 9915

rw------ 1 squser4 seaquest 113142116 Jun 8 18:51 jmapdump-9915.0
rw------ 1 squser4 seaquest 341615426 Jun 8 19:58 jmapdump-9915.13

and here is the initial jmap-9915.0 old space:

PS Old Generation
capacity = 1407713280 (1342.5MB)
used = 100979864 (96.3019027709961MB)
free = 1306733416 (1246.198097229004MB)
7.17332609094943% used

and then jmap-9915.13

PS Old Generation
capacity = 571998208 (545.5MB)
used = 250169256 (238.57999420166016MB)
free = 321828952 (306.92000579833984MB)

it grows about 12Meg every 5 minutes.