Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-5599

HttpReplicator uses a lot of CPU for large files

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 4.7.1
    • 4.8.1, 4.9, 6.0
    • modules/replicator
    • None
    • New, Patch Available

    Description

      The method responseInputStream of HttpClientBase wraps an InputStream in order to close it when it is done reading. However, the wrapper only overwrites the single-byte read() method, every other method is delegated to its parent (java.io.InputStream). Therefore, the more efficient read-methods like read(byte[] b) are all implemented by reading one byte after the other.

      In my test, it took 20 minutes to copy an index of 38 GB. With the provided small patch, this was reduced to less than 10 minutes.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            shaie Shai Erera
            christophk Christoph Kaser
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment