[GEODE-7370] ClassGraph library causes large memory leak in Geode after adding geode-log4j - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 1.11.0
Component/s: management
Labels:
None

Description

According to the source code and documentation, io.github.classgraph.ScanResult must be closed to avoid memory and file leaks.

I identified a large memory leak that apparently after adding geode-log4j. Surprisingly the cause ended up being an instance of ScanResult which is used by the management code to search the classpath for user classes that implement geode functions or gfsh commands.

Geode currently depends on ClassGraph version 4.0.6, but the library is up to version 4.8.52 (averaging around a dozen releases per month).

JVM bytes in use by io.github.classgraph after closing all ScanResult instances:

Before geode-log4j with ClassGraph 4.0.6: 20,488 bytes
Before geode-log4j with ClassGraph 4.8.52: 1,056 bytes
After geode-log4j with ClassGraph 4.0.6: 56,753,008 bytes
After geode-log4j with ClassGraph 4.8.52: 1,056 bytes

Given the above results of my testing, I believe we need to keep this dependency up-to-date and upgrade to 4.8.52.

More detailed notes below:

Git version shas that were tested:

Before: 802687154131af16d350bf39d152feb4685ba7e6 (sha before geode-log4j)
Before-w/upgrade: Before w/ ClassGraph upgraded from 4.0.6 to 4.8.52
<< Geode-log4j: efc2362d2bae0877a427ce2c29beae94118d6567 (geode-log4j) >>
After: dffcb9446aef09c7bf6e626121f4d2ec5c74586f (async alerts)
After-w/upgrade: After w/ ClassGraph upgraded from 4.0.6 to 4.8.52

Java process memory sizes:

Before:

    ChildVMs
    22297  java         0.0    00:07.51 32     1    104   252M   0B     0B     22220 22286 sleeping *0[1]            0.00000 0.00000    501
    22295  java         0.0    00:07.72 32     1    104   259M   0B     0B     22220 22286 sleeping *0[1]            0.00000 0.00000    501
    22291  java         0.1    00:11.66 32     1    104   291M   0B     0B     22220 22286 sleeping *0[1]            0.00000 0.00000    501
    22289  java         0.1    00:11.68 32     1    104   300M   0B     0B     22220 22286 sleeping *0[1]            0.00000 0.00000    501
    22287  java         0.5    00:07.67 64     1    168   276M+  0B     0B     22220 22286 sleeping *0[1]            0.00000 0.00000    501
    GradleWorkerMain
    22286  java         0.4    00:03.55 67     1    174   184M+  0B     0B     22220 22220 sleeping *0[1]            0.00000 0.00000    501
    GradleWrapperMain
    22173  java         0.1    00:01.63 28     1    96    88M+   0B     0B     22173 86991 sleeping *0[1]            0.00000 0.00000    501
    GradleDaemon
    22220  java         0.6    00:52.27 81     1    201   814M   0B     0B     22220 22173 sleeping *0[1]            0.00000 0.00000    501

Before-w/upgrade:

    ChildVMs
    25789  java         0.0  00:06.93 31     1    101   257M   0B     0B     25714 25779 sleeping *0[1]           0.00000 0.00000    501
    25788  java         0.1  00:06.93 31     1    101   259M+  0B     0B     25714 25779 sleeping *0[1]           0.00000 0.00000    501
    25784  java         0.0  00:10.37 31     1    101   286M   0B     0B     25714 25779 sleeping *0[1]           0.00000 0.00000    501
    25782  java         0.1  00:11.68 31     1    101   278M   0B     0B     25714 25779 sleeping *0[1]           0.00000 0.00000    501
    25780  java         0.3  00:07.84 63/1   1    165   308M+  0B     0B     25714 25779 running  *0[1]           0.00000 0.00000    501
    GradleWorkerMain
    25779  java         0.1  00:03.39 60     1    160   183M   0B     0B     25714 25714 sleeping *0[1]           0.00000 0.00000    501
    GradleDaemon
    25714  java         0.8  00:48.79 79     1    197   793M   0B     0B     25714 25667 sleeping *0[1]           0.00000 0.00000    501
    GradleWrapperMain
    25667  java         0.2  00:02.16 28     1    96    101M+  0B     0B     25667 86991 sleeping *0[1]           0.00000 0.00000    501

After:

    ChildVMs
    21056  java         0.1  00:06.67 31     1    101   256M   0B     0B     20988 21049 sleeping *0[1]            0.00000 0.00000    501
    21055  java         0.1  00:06.82 31     1    101   264M+  0B     0B     20988 21049 sleeping *0[1]            0.00000 0.00000    501
    21053  java         0.1  00:15.95 31     1    101   373M   0B     0B     20988 21049 sleeping *0[1]            0.00000 0.00000    501
    21051  java         0.3  00:14.96 31     1    101   363M+  0B     0B     20988 21049 sleeping *0[1]            0.00000 0.00000    501
    21050  java         0.5  00:12.07 63     1    165   343M+  0B     0B     20988 21049 sleeping *0[1]            0.00000 0.00000    501
    GradleWorkerMain
    21049  java         0.2  00:06.99 60     1    160   321M+  0B     0B     20988 20988 sleeping *0[1]            0.00000 0.00000    501
    GradleWrapperMain
    20941  java         0.2  00:02.14 28     1    96    101M+  0B     0B     20941 86991 sleeping *0[1]            0.00000 0.00000    501
    GradleDaemon
    20988  java         0.8  00:53.39 81     1    202   963M   0B     0B     20988 20941 sleeping *0[1]            0.00000 0.00000    501

After-w/upgrade:

    ChildVMs
    24298  java         0.1  00:06.29 32     1    104   250M   0B     0B     24231 24291 sleeping *0[1]            0.00000 0.00000    501
    24297  java         0.1  00:06.59 32     1    104   262M   0B     0B     24231 24291 sleeping *0[1]            0.00000 0.00000    501
    24295  java         0.1  00:14.97 32     1    104   362M   0B     0B     24231 24291 sleeping *0[1]            0.00000 0.00000    501
    24293  java         0.2  00:15.44 32     1    104   376M+  0B     0B     24231 24291 sleeping *0[1]            0.00000 0.00000    501
    24292  java         0.7  00:10.36 64     1    168   327M   0B     0B     24231 24291 sleeping *0[1]            0.00000 0.00000    501
    GradleWorkerMain
    24291  java         0.7  00:06.07 67     1    174   320M+  0B     0B     24231 24231 sleeping *0[1]            0.00000 0.00000    501
    GradleWrapperMain
    24184  java         0.3  00:01.39 28     1    96    89M+   0B     0B     24184 86991 sleeping *0[1]            0.00000 0.00000    501
    GradleDaemon
    24231  java         1.0  00:46.21 81     1    201   805M   0B     0B     24231 24184 sleeping *0[1]            0.00000 0.00000    501

Biggest objects - Dominators comparison (retained size in bytes):

  Before:
    sun.misc.Launcher$AppClassLoader - 883,984
  Before-w/upgrade:
    sun.misc.Launcher$AppClassLoader - 915,384
  After:
    io.github.classgraph.ScanResult - 11,373,088
  After-w/upgrade:
    sun.misc.Launcher$AppClassLoader - 1,256,904

Class and package comparison (retained size in bytes):

org.apache.geode.internal.logging -
  Before: 253,520
  Before-w/upgrade: 253,520
  After: 33,576
  After-w/upgrade: 21,152
org.apache.geode.logging -
  Before: 0
  Before-w/upgrade: 0
  After: 233,104
  After-w/upgrade: 231,968
org.apache.logging.log4j -
  Before: 581,184
  Before-w/upgrade:  581,184
  After: 605,904
  After-w/upgrade: 584,576
java.util.logging -
  Before: 45,632
  Before-w/upgrade: 45,632
  After: 45,632
  After-w/upgrade: 45,632

org.apache.geode.internal.cache -
  Before: 1,034,624
  Before-w/upgrade: 1,034,560
  After: 1,065,312
  After-w/upgrade: 1,034,224
org.apache.geode.management -
  Before: 9,592
  Before-w/upgrade: 9,592
  After: 28,408
  After-w/upgrade: 9,592
org.apache.geode.distributed -
  Before: 224,960
  Before-w/upgrade: 224,408
  After: 256,272
  After-w/upgrade: 223,720

io.github.classgraph -
  Before: 20,488
  Before-w/upgrade: 1,056
  After: 56,753,008
  After-w/upgrade: 1,056

java -
  Before: 38,697,928
  Before-w/upgrade: 59,048,552
  After: 143,295,448
  After-w/upgrade: 52,397,344
sun -
  Before: 2,464,792
  Before-w/upgrade: 2,593,712
  After: 3,359,248
  After-w/upgrade: 3,126,280

Class loader comparison:

System class loader -
  Before: 356,651 objects, 38,697,928 bytes, 4,295 classes, 3,028 classes w/o instances
  Before-w/upgrade: 883,377 objects, 59,048,552 bytes, 4,344 classes, 3,077 classes w/o instances
  After: 1,802,231 objects, 143,295,448 bytes, 8,715 classes, 7,191 classes w/o instances
  After-w/upgrade: 712,102 objects, 52,397,344 bytes, 8,765 classes, 7,461 classes w/o instances
AppClassLoader -
  Before: 11,275 objects, 7,020,720 bytes, 2,885 classes, 1,896 classes w/o instances
  Before-w/upgrade: 10,467 objects, 7,068,488 bytes, 2,921 classes, 1,943 classes w/o instances
  After: 20,336 objects, 64,013,592 bytes, 2,929 classes, 1,451 classes w/o instances
  After-w/upgrade: 10,988 objects, 7,273,232 bytes, 2,980 classes, 1,980 classes w/o instances
Geode ClassPathLoader$1 -
  Before: 2 objects, 130,480 bytes, 1 classes, 0 classes w/o instances
  Before-w/upgrade: 2 objects, 130,480 bytes, 1 classes, 0 classes w/o instances
  After: 2 Objects, 130,680 bytes, 1 classes, 0 classes w/o instances
  After-w/upgrade: 2 objects, 130,680 bytes, 1 classes, 0 classes w/o instances

Reachability comparison (retained size in bytes):

Objects unreachable from GC roots, but not yet collected -
  Before: 26,669,744
  Before-w/upgrade: 46,758,496
  After: 112,345,880
  After-w/upgrade: 35,093,072
Objects reachable from GC roots via strong references -
  Before: 12,028,184
  Before-w/upgrade: 12,290,056
  After: 30,949,568
  After-w/upgrade: 17,304,272
Objects pending finalization (finalizer queue objects unreachable via strong references) -
  Before: 241,776
  Before-w/upgrade: 252,192
  After: 13,564,760
  After-w/upgrade: 295,160
Objects reachable from GC roots via weak and/or soft references only -
  Before: 91,048
  Before-w/upgrade: 106,488
  After: 586,616
  After-w/upgrade: 51,616

Attachments

Issue Links

links to

GitHub Pull Request #4237

Activity

People

Assignee:: Kirk Lund

Reporter:: Kirk Lund

Votes:: 0 Vote for this issue

Watchers:: 3 Start watching this issue

Dates

Created:: 28/Oct/19 19:22

Updated:: 30/Dec/19 18:50

Resolved:: 01/Nov/19 20:20

Time Tracking

Estimated:

Not Specified

Remaining:

Logged:

0.5h