Solr
  1. Solr
  2. SOLR-797

Construct EmbeddedSolrServer response without serializing/parsing

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.3
    • Fix Version/s: 4.9, 5.0
    • Component/s: clients - java
    • Labels:
      None

      Description

      Currently, the EmbeddedSolrServer serializes the response and reparses in order to create the final NamedList response. From the comment in EmbeddedSolrServer.java, the goal is to:

      • convert the response directly into a named list
      1. SOLR-797.patch
        2 kB
        Gregg Donovan
      2. SOLR-797.patch
        11 kB
        Andrey Klochkov
      3. SOLR-797.patch
        6 kB
        Jonathan Lee

        Issue Links

          Activity

          Hide
          Jonathan Lee added a comment -

          This patch contains a first stab at transforming the NamedList without serializing it then parsing it from the serialized form.

          From what I can tell, all the fields (headers, facets, spelling, etc) returned from the handler in the response is valid for output except that references to actual documents need to be resolved. This patch borrows code from NamedListCodec.java and BinaryResponseWriter.java to resolve the documents.

          Show
          Jonathan Lee added a comment - This patch contains a first stab at transforming the NamedList without serializing it then parsing it from the serialized form. From what I can tell, all the fields (headers, facets, spelling, etc) returned from the handler in the response is valid for output except that references to actual documents need to be resolved. This patch borrows code from NamedListCodec.java and BinaryResponseWriter.java to resolve the documents.
          Hide
          Shalin Shekhar Mangar added a comment -

          Marking for 1.5

          Show
          Shalin Shekhar Mangar added a comment - Marking for 1.5
          Hide
          Andrey Klochkov added a comment -

          Sometime ago I made such an improvement and now decided to share it. Looks like this is the right issue. Attaching my patch, please use it if it looks better. We use our patched version without problems for several months, so it's tested in at some level.

          Show
          Andrey Klochkov added a comment - Sometime ago I made such an improvement and now decided to share it. Looks like this is the right issue. Attaching my patch, please use it if it looks better. We use our patched version without problems for several months, so it's tested in at some level.
          Hide
          Hoss Man added a comment -

          Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email...

          http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E

          Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed.

          A unique token for finding these 240 issues in the future: hossversioncleanup20100527

          Show
          Hoss Man added a comment - Bulk updating 240 Solr issues to set the Fix Version to "next" per the process outlined in this email... http://mail-archives.apache.org/mod_mbox/lucene-dev/201005.mbox/%3Calpine.DEB.1.10.1005251052040.24672@radix.cryptio.net%3E Selection criteria was "Unresolved" with a Fix Version of 1.5, 1.6, 3.1, or 4.0. email notifications were suppressed. A unique token for finding these 240 issues in the future: hossversioncleanup20100527
          Hide
          Robert Muir added a comment -

          Bulk move 3.2 -> 3.3

          Show
          Robert Muir added a comment - Bulk move 3.2 -> 3.3
          Hide
          Robert Muir added a comment -

          3.4 -> 3.5

          Show
          Robert Muir added a comment - 3.4 -> 3.5
          Hide
          Hoss Man added a comment -

          Bulk of fixVersion=3.6 -> fixVersion=4.0 for issues that have no assignee and have not been updated recently.

          email notification suppressed to prevent mass-spam
          psuedo-unique token identifying these issues: hoss20120321nofix36

          Show
          Hoss Man added a comment - Bulk of fixVersion=3.6 -> fixVersion=4.0 for issues that have no assignee and have not been updated recently. email notification suppressed to prevent mass-spam psuedo-unique token identifying these issues: hoss20120321nofix36
          Hide
          Steve Rowe added a comment -

          Bulk move 4.4 issues to 4.5 and 5.0

          Show
          Steve Rowe added a comment - Bulk move 4.4 issues to 4.5 and 5.0
          Hide
          Gregg Donovan added a comment -

          I'm interested in this as well. We had an custom API that was similar to the attached patch. When we switched to EmbeddedSolrServer we noticed an increase in time spent deserializing the Solr response, memory allocated, and GC spikiness.

          One issue with the current EmbeddedSolrServer code is that it starts with ByteArrayOutputStream of 32 bytes and resizes repeatedly it to fit the results. We have large responses and we notice the GC hit. We experimented with a ThreadLocal<ByteBuffer>, but avoiding serializing and parsing altogether for EmbeddedSolrServer seems like an even better idea.

          If there's interest, we'd be happy to revive/update/test this patch.

          Show
          Gregg Donovan added a comment - I'm interested in this as well. We had an custom API that was similar to the attached patch. When we switched to EmbeddedSolrServer we noticed an increase in time spent deserializing the Solr response, memory allocated, and GC spikiness. One issue with the current EmbeddedSolrServer code is that it starts with ByteArrayOutputStream of 32 bytes and resizes repeatedly it to fit the results. We have large responses and we notice the GC hit. We experimented with a ThreadLocal<ByteBuffer>, but avoiding serializing and parsing altogether for EmbeddedSolrServer seems like an even better idea. If there's interest, we'd be happy to revive/update/test this patch.
          Hide
          Gregg Donovan added a comment -

          After some experimentation, it seems that reproducing exactly what happens when BinaryResponseWriter serializes and deserializes the graph without actually doing either is quite a bit of work.

          Attached is a less ambitious patch that just attempts to reduce the amount of allocation of byte[] involved in [de]serializing.

          Show
          Gregg Donovan added a comment - After some experimentation, it seems that reproducing exactly what happens when BinaryResponseWriter serializes and deserializes the graph without actually doing either is quite a bit of work. Attached is a less ambitious patch that just attempts to reduce the amount of allocation of byte[] involved in [de] serializing.
          Hide
          Uwe Schindler added a comment -

          Move issue to Solr 4.9.

          Show
          Uwe Schindler added a comment - Move issue to Solr 4.9.

            People

            • Assignee:
              Unassigned
              Reporter:
              Jonathan Lee
            • Votes:
              3 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:

                Development