[TIKA-3237] Great optimization in ForkParser

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.24.1
• Fix Version/s: 2.0.0
• Component/s: core

Environment

Windows 10, Oracle JDK 1.8.0_261 x64, 24 cores

• Assignee: Luís Filipe Nassif
• Reporter: Luís Filipe Nassif
• Created: 26/Nov/20 18:22
• Updated: 12/Apr/21 12:59
• Resolved: 26/Nov/20 20:25

Description

There is a huge overhead in ForkParser ContentHandlerProxy and ContentHandlerResource read/write char[]/string methods. A simple change to not loop reading/writing each char of strings can result in speed ups of several orders of magnitude.

 

Tests with ~10k small RFC822 files (22MB), 24 parsing threads, have shown parse times below:

ForkParser off: 4s

ForkParser on: 30min

ForkParser patched: 17s

 

Also, there is a bug when running at least with Oracle JDK8 that, if a String greater than 65535 chars is written, a UTFDataFormatException is thrown from DataOutputStream.writeUTF(String) method.