Lucene - Core
  1. Lucene - Core
  2. LUCENE-5599

HttpReplicator uses a lot of CPU for large files

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 4.7.1
    • Fix Version/s: 4.8.1, 4.9, 6.0
    • Component/s: modules/replicator
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      The method responseInputStream of HttpClientBase wraps an InputStream in order to close it when it is done reading. However, the wrapper only overwrites the single-byte read() method, every other method is delegated to its parent (java.io.InputStream). Therefore, the more efficient read-methods like read(byte[] b) are all implemented by reading one byte after the other.

      In my test, it took 20 minutes to copy an index of 38 GB. With the provided small patch, this was reduced to less than 10 minutes.

        Activity

        Hide
        Shawn Heisey added a comment -

        General question, not directed at anyone specific: Does Solr replication suffer from a similar problem? I haven't used replication in quite some time, but it is an integral part of SolrCloud node recovery and replication in non-cloud setups is a widely used feature.

        Show
        Shawn Heisey added a comment - General question, not directed at anyone specific: Does Solr replication suffer from a similar problem? I haven't used replication in quite some time, but it is an integral part of SolrCloud node recovery and replication in non-cloud setups is a widely used feature.
        Hide
        Christoph Kaser added a comment -

        I don't think so. As far as I know, lucene replication and solr replication don't share any code at the moment, so this should only affect lucene.

        Show
        Christoph Kaser added a comment - I don't think so. As far as I know, lucene replication and solr replication don't share any code at the moment, so this should only affect lucene.
        Hide
        ASF subversion and git services added a comment -

        Commit 1587930 from shaie@apache.org in branch 'dev/trunk'
        [ https://svn.apache.org/r1587930 ]

        LUCENE-5599: HttpClientBase did not properly delegate to wrapped InputStream

        Show
        ASF subversion and git services added a comment - Commit 1587930 from shaie@apache.org in branch 'dev/trunk' [ https://svn.apache.org/r1587930 ] LUCENE-5599 : HttpClientBase did not properly delegate to wrapped InputStream
        Hide
        ASF subversion and git services added a comment -

        Commit 1587931 from shaie@apache.org in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1587931 ]

        LUCENE-5599: HttpClientBase did not properly delegate to wrapped InputStream

        Show
        ASF subversion and git services added a comment - Commit 1587931 from shaie@apache.org in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1587931 ] LUCENE-5599 : HttpClientBase did not properly delegate to wrapped InputStream
        Hide
        Shai Erera added a comment -

        Thanks Christoph, this is really a silly bug, nice catch!

        I've committed to trunk and 4x.

        Show
        Shai Erera added a comment - Thanks Christoph, this is really a silly bug, nice catch! I've committed to trunk and 4x.
        Hide
        ASF subversion and git services added a comment -

        Commit 1592584 from Michael McCandless in branch 'dev/branches/lucene_solr_4_8'
        [ https://svn.apache.org/r1592584 ]

        LUCENE-5599, LUCENE-5600, LUCENE-5628: backport to 4.8.x

        Show
        ASF subversion and git services added a comment - Commit 1592584 from Michael McCandless in branch 'dev/branches/lucene_solr_4_8' [ https://svn.apache.org/r1592584 ] LUCENE-5599 , LUCENE-5600 , LUCENE-5628 : backport to 4.8.x

          People

          • Assignee:
            Shai Erera
            Reporter:
            Christoph Kaser
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development