[FILEUPLOAD-288] Linux dentry cache memory leak due to inappropriate temporary file detection

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Nightly Builds, 1.4

Environment

OS version: CentOS release 6.7 (Final)
Kenel: 2.6.32-573.26.1.el6.x86_64
JVM: Oracle 1.8.0_102-b14
Tomcat: 8.0.33.0
commons-fileupload : 1.3.3

• Assignee: Unassigned
• Reporter: fangwentong
• Created: 23/Nov/17 05:09
• Updated: 05/Feb/19 14:11
• Resolved: 23/Nov/17 20:44

Description

We use commons-fileupload:1.3.3 to handle multipart request, param file-size-threshold was set to 200K. Our API handle multipart request in high QPS, the size of request body is about 20KB, which means those parts was saved in memory, not written to disk temporary file.

But, in our production environment, size of dentry cache grows up in high speed. here are some analysis:

in strace result, plenty of stat system call invoked for tmp files which not exists.

[pid 111310] <... stat resumed> {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0
[pid 57844] <... stat resumed> 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624923.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624922.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624921.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624920.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624919.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624918.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 69798] stat("/tmp",  <unfinished ...>
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624917.tmp",  <unfinished ...>
[pid 69798] <... stat resumed> {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0
[pid 57844] <... stat resumed> 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624916.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624915.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624914.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624913.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624912.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624911.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624910.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624909.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624908.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624907.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624906.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624905.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624904.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624903.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624902.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624901.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624900.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624899.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624898.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624897.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624896.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624895.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624894.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624893.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624892.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624891.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624890.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624889.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624888.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624887.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624886.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624885.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624884.tmp", 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
[pid 78454] --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=57810, si_uid=605} ---

in slabtop result , dentry cache taken up 22GB+ memory.

 Active / Total Objects (% used)    : 233715492 / 234153507 (99.8%)
 Active / Total Slabs (% used)      : 7858579 / 7858606 (100.0%)
 Active / Total Caches (% used)     : 108 / 200 (54.0%)
 Active / Total Size (% used)       : 29322726.34K / 29380106.43K (99.8%)
 Minimum / Average / Maximum Object : 0.02K / 0.12K / 4096.00K

  OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
116095303 116095295  26%    0.06K 1967717       59   7870868K size-64
116090260 116090260  26%    0.19K 5804513       20  23218052K dentry

finally, I think the root cause is：in temporary file cleanup policy defined at DiskFileItem#finalize, call outputFile.exists() for every File, even if the part not write to disk.

    @Override
    protected void finalize() {
        if (dfos == null) {
            return;
        }
        File outputFile = dfos.getFile();

        if (outputFile != null && outputFile.exists()) {
            outputFile.delete();
        }
    }

Issue Links

• links to: GitHub Pull Request #12