Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-9147

avoid expensive byte[] resize in EmbeddedSolrServer

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 6.1, 7.0
    • Component/s: Server
    • Labels:
      None
    • Flags:
      Patch

      Description

      This issue makes modest step toward EmbeddedSolrServer efficiency.
      It replaces java.io.ByteArrayOutputStream which has quite expensive resizes with incrementally growing BAOS from commons-io 2.5.

      Note

      There is no expectation for performance gain in case of StreamingResponseCallback.

      1. SOLR-9147.patch
        6 kB
        Mikhail Khludnev
      2. SOLR-9147.patch
        6 kB
        Mikhail Khludnev

        Issue Links

          Activity

          Hide
          mkhludnev Mikhail Khludnev added a comment -

          I'm going to commit it soon.
          Discussion had place at SOLR-797, SOLR-6578.

          Show
          mkhludnev Mikhail Khludnev added a comment - I'm going to commit it soon. Discussion had place at SOLR-797 , SOLR-6578 .
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          also need to remove commons-io-unsafe-2.4.jar from solr/common-build.xml it seems lucene's twin goes fine without it.

          Show
          mkhludnev Mikhail Khludnev added a comment - also need to remove commons-io-unsafe-2.4.jar from solr/common-build.xml it seems lucene's twin goes fine without it.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 3ea12f2be4f39a5bafdd8c48b113d9753ae637ad in lucene-solr's branch refs/heads/master from Mikhail Khludnev
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3ea12f2 ]

          SOLR-9147: using BAOS from commons io 2.5 in EmbeddedSolrServer to avoid array resize.

          Show
          jira-bot ASF subversion and git services added a comment - Commit 3ea12f2be4f39a5bafdd8c48b113d9753ae637ad in lucene-solr's branch refs/heads/master from Mikhail Khludnev [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3ea12f2 ] SOLR-9147 : using BAOS from commons io 2.5 in EmbeddedSolrServer to avoid array resize.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit bbac85c35aba3155add140ffbcb607f9780d722b in lucene-solr's branch refs/heads/branch_6x from Mikhail Khludnev
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=bbac85c ]

          SOLR-9147: using BAOS from commons io 2.5 in EmbeddedSolrServer to avoid array resize.

          Show
          jira-bot ASF subversion and git services added a comment - Commit bbac85c35aba3155add140ffbcb607f9780d722b in lucene-solr's branch refs/heads/branch_6x from Mikhail Khludnev [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=bbac85c ] SOLR-9147 : using BAOS from commons io 2.5 in EmbeddedSolrServer to avoid array resize.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          oops.. will move it to Optimizations section in CHANGES.txt shortly

          Show
          mkhludnev Mikhail Khludnev added a comment - oops.. will move it to Optimizations section in CHANGES.txt shortly
          Hide
          thetaphi Uwe Schindler added a comment -

          Why did you remove commons-io from forbiddenapis? I will readd them.

          Unfortunately the new version was released half an hour ago, unfortunately not yet with commons-io-2.5 support. So I'd go with commons-io-2.4 signatures for now.

          Show
          thetaphi Uwe Schindler added a comment - Why did you remove commons-io from forbiddenapis? I will readd them. Unfortunately the new version was released half an hour ago, unfortunately not yet with commons-io-2.5 support. So I'd go with commons-io-2.4 signatures for now.
          Hide
          thetaphi Uwe Schindler added a comment -

          I restored forbidden-apis. Next time please ask before removing important stuff without a workaround!

          Show
          thetaphi Uwe Schindler added a comment - I restored forbidden-apis. Next time please ask before removing important stuff without a workaround!
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 3fb69edead6721146997a13a3e0b2c8ca918abef in lucene-solr's branch refs/heads/master from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3fb69ed ]

          LUCENE-7296, SOLR-9147: Fix Maven build

          Show
          jira-bot ASF subversion and git services added a comment - Commit 3fb69edead6721146997a13a3e0b2c8ca918abef in lucene-solr's branch refs/heads/master from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3fb69ed ] LUCENE-7296 , SOLR-9147 : Fix Maven build
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit b385a536f3a94e7a95418bc5ecf847fc1a63fd18 in lucene-solr's branch refs/heads/branch_6x from Uwe Schindler
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b385a53 ]

          LUCENE-7296, SOLR-9147: Fix Maven build

          Show
          jira-bot ASF subversion and git services added a comment - Commit b385a536f3a94e7a95418bc5ecf847fc1a63fd18 in lucene-solr's branch refs/heads/branch_6x from Uwe Schindler [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b385a53 ] LUCENE-7296 , SOLR-9147 : Fix Maven build
          Hide
          thetaphi Uwe Schindler added a comment -

          I opened issue at forbiddenapis: https://github.com/policeman-tools/forbidden-apis/issues/102

          Unfortunatley this issue was a bit too late for the release! You should have pinged me directly, now we must live with a hack in both the Ant and Maven builds.

          Show
          thetaphi Uwe Schindler added a comment - I opened issue at forbiddenapis: https://github.com/policeman-tools/forbidden-apis/issues/102 Unfortunatley this issue was a bit too late for the release! You should have pinged me directly, now we must live with a hack in both the Ant and Maven builds.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit d8a6bf8a052f3d7cd3ee8d254fa501b09b8719be in lucene-solr's branch refs/heads/master from Mikhail Khludnev
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d8a6bf8 ]

          SOLR-9147: trying to fix the build

          Show
          jira-bot ASF subversion and git services added a comment - Commit d8a6bf8a052f3d7cd3ee8d254fa501b09b8719be in lucene-solr's branch refs/heads/master from Mikhail Khludnev [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=d8a6bf8 ] SOLR-9147 : trying to fix the build
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 02a31ebc8461a2e829b126715297e8fa51bb83da in lucene-solr's branch refs/heads/branch_6x from Mikhail Khludnev
          [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=02a31eb ]

          SOLR-9147: trying to fix the build

          Show
          jira-bot ASF subversion and git services added a comment - Commit 02a31ebc8461a2e829b126715297e8fa51bb83da in lucene-solr's branch refs/heads/branch_6x from Mikhail Khludnev [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=02a31eb ] SOLR-9147 : trying to fix the build
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          it causes the failure
          https://builds.apache.org/job/Lucene-Solr-Tests-master/1162/console

          check-working-copy:
          ...
          /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:608: Source checkout is modified!!! Offending files:
          * solr/licenses/commons-io-2.5.jar.sha1
          

          I'm trying to fix it quite unconsciously by commits below.

          Show
          mkhludnev Mikhail Khludnev added a comment - it causes the failure https://builds.apache.org/job/Lucene-Solr-Tests-master/1162/console check-working-copy: ... /x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:608: Source checkout is modified!!! Offending files: * solr/licenses/commons-io-2.5.jar.sha1 I'm trying to fix it quite unconsciously by commits below.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          Uwe Schindler I decided to remove it because build didn't fail locally, and I see that it's not used in Lucene. Anyway, I apology for breaking things, which I don't understand (but it's what I'm doing all of my life).

          Also, how to fix:

           
          build.xml:608: Source checkout is modified!!! Offending files:
          * solr/licenses/commons-io-2.5.jar.sha1
          
          Show
          mkhludnev Mikhail Khludnev added a comment - Uwe Schindler I decided to remove it because build didn't fail locally, and I see that it's not used in Lucene. Anyway, I apology for breaking things, which I don't understand (but it's what I'm doing all of my life). Also, how to fix: build.xml:608: Source checkout is modified!!! Offending files: * solr/licenses/commons-io-2.5.jar.sha1
          Hide
          mkhludnev Mikhail Khludnev added a comment - - edited

          for the reference, by Uwe Schindler:

          Since we changed to Git, you may better use Linux to generate the checksums. Currently there seems to be a problem with line endings! I wanted to open an issue about that already, but changes to the sha1 are seldom… So be sure to check that you only see a difference in this one file after “ant jar-checksums”.

          Show
          mkhludnev Mikhail Khludnev added a comment - - edited for the reference, by Uwe Schindler : Since we changed to Git, you may better use Linux to generate the checksums. Currently there seems to be a problem with line endings! I wanted to open an issue about that already, but changes to the sha1 are seldom… So be sure to check that you only see a difference in this one file after “ant jar-checksums”.
          Hide
          mkhludnev Mikhail Khludnev added a comment -

          Uwe Schindler, can you point me on exact failure, I want understand what I broke and to be able to check it locally next time.

          Show
          mkhludnev Mikhail Khludnev added a comment - Uwe Schindler , can you point me on exact failure, I want understand what I broke and to be able to check it locally next time.
          Hide
          thetaphi Uwe Schindler added a comment - - edited

          What failure do you mean? There are 2 probs here:

          • fobiddenapis is used to find calls to "broken functions" in the JDK and also commons-io (like Readers without Charset,...). Lucene and Solr also have a list of other signatures that are disallowed (like creating Threads without a name, using log4j instead of slf4j,...). The problem with the update of commons-io was, that there is currently no signature list for the 2.5 version of commons-io bundled with the checking tool. So instead of disabling, I re-added the signatures of the 2.4 version, which should be fine for now. I opened an issue to add support for commons-io-2.5: https://github.com/policeman-tools/forbidden-apis/issues/102
          • The Jar Checksum failure on Jenkins is comming from the following: Jenkins uses the "jar-checksums" task and recalculates all checksums and this task also writes them to disk. In the final check of Jenkins (after running all tests, recalculating checksums,...) the Git checkout is checked for modifications. If the checksums aren't correct, this fails. This is what happened. The problem was that you created the checksum files by hand, not with the ant task, so there was just the newline difference.
          Show
          thetaphi Uwe Schindler added a comment - - edited What failure do you mean? There are 2 probs here: fobiddenapis is used to find calls to "broken functions" in the JDK and also commons-io (like Readers without Charset,...). Lucene and Solr also have a list of other signatures that are disallowed (like creating Threads without a name, using log4j instead of slf4j,...). The problem with the update of commons-io was, that there is currently no signature list for the 2.5 version of commons-io bundled with the checking tool. So instead of disabling, I re-added the signatures of the 2.4 version, which should be fine for now. I opened an issue to add support for commons-io-2.5: https://github.com/policeman-tools/forbidden-apis/issues/102 The Jar Checksum failure on Jenkins is comming from the following: Jenkins uses the "jar-checksums" task and recalculates all checksums and this task also writes them to disk. In the final check of Jenkins (after running all tests, recalculating checksums,...) the Git checkout is checked for modifications. If the checksums aren't correct, this fails. This is what happened. The problem was that you created the checksum files by hand, not with the ant task, so there was just the newline difference.
          Hide
          thetaphi Uwe Schindler added a comment - - edited

          ant precommit does the same like jenkins. So you should call that before committing, especially if you change dependencies in the build or shuffle modules around.

          This task does a lot of checks, including that all licenses are there, the source code contains no tabs instead of spaces, the above forbidden-apis checks, it also builds javadocs and checks the Javadocs for completeness and broken links. This may take a while, but you can go and fetch a beer in the meantime

          Show
          thetaphi Uwe Schindler added a comment - - edited ant precommit does the same like jenkins. So you should call that before committing, especially if you change dependencies in the build or shuffle modules around. This task does a lot of checks, including that all licenses are there, the source code contains no tabs instead of spaces, the above forbidden-apis checks, it also builds javadocs and checks the Javadocs for completeness and broken links. This may take a while, but you can go and fetch a beer in the meantime
          Hide
          thetaphi Uwe Schindler added a comment - - edited

          MacOSX is also fine. This note was about Windows, where ant jar-checksums produces differences. The reason is that the default Git configuration has no NL policy for files with "sha1" as extension. Maybe we should add some git config properties to the checkout. Does anybody know how to do this? like ".gitignore"?

          Show
          thetaphi Uwe Schindler added a comment - - edited MacOSX is also fine. This note was about Windows, where ant jar-checksums produces differences. The reason is that the default Git configuration has no NL policy for files with "sha1" as extension. Maybe we should add some git config properties to the checkout. Does anybody know how to do this? like ".gitignore"?
          Hide
          n0v1 Chris F added a comment -

          I think you can use a .gitattributes file to specify the desired line endings per file extension. See https://help.github.com/articles/dealing-with-line-endings/#per-repository-settings

          Show
          n0v1 Chris F added a comment - I think you can use a .gitattributes file to specify the desired line endings per file extension. See https://help.github.com/articles/dealing-with-line-endings/#per-repository-settings

            People

            • Assignee:
              mkhludnev Mikhail Khludnev
              Reporter:
              mkhludnev Mikhail Khludnev
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development