Uploaded image for project: 'Tika'
  1. Tika
  2. TIKA-3237

Great optimization in ForkParser

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.24.1
    • Fix Version/s: 2.0.0
    • Component/s: core
    • Labels:
      None
    • Environment:

      Windows 10, Oracle JDK 1.8.0_261 x64, 24 cores

      Description

      There is a huge overhead in ForkParser ContentHandlerProxy and ContentHandlerResource read/write char[]/string methods. A simple change to not loop reading/writing each char of strings can result in speed ups of several orders of magnitude.

       

      Tests with ~10k small RFC822 files (22MB), 24 parsing threads, have shown parse times below:

      ForkParser off: 4s

      ForkParser on: 30min

      ForkParser patched: 17s

       

      Also, there is a bug when running at least with Oracle JDK8 that, if a String greater than 65535 chars is written, a UTFDataFormatException is thrown from DataOutputStream.writeUTF(String) method.

        Attachments

          Activity

            People

            • Assignee:
              lfcnassif Luís Filipe Nassif
              Reporter:
              lfcnassif Luís Filipe Nassif

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment