[TIKA-2802] Out of memory issues when extracting large files (pst)

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.20, 1.19.1
• Component/s: parser

Environment

Reproduced on Windows 2012 R2 and Ubuntu 18.04.

Java: jdk1.8.0_151

 

• Assignee: Unassigned
• Reporter: Caleb Ott
• Created: 27/Dec/18 15:55
• Updated: 16/May/19 06:40
• Resolved: 31/Jan/19 14:01

Description

I have an application that extracts text from multiple files on a file share. I've been running into issues with the application running out of memory (~26g dedicated to the heap).

I found in the heap dumps there is a "fDTDDecl" buffer which is creating very large char arrays and never releasing that memory. In the picture you can see the heap dump with 4 SAXParsers holding onto a large chunk of memory. The fourth one is expanded to show it is all being held by the "fDTDDecl" field. This dump is from a scaled down execution (not a 26g heap).

It looks like that DTD field should never be that large, I'm wondering if this is a bug with xerces instead? I can easily reproduce the issue by attempting to extract text from large .pst files.