Solr
  1. Solr
  2. SOLR-5875

QueryComponent.mergeIds() unmarshals all docs' sort field values once per doc instead of once per shard

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 4.7
    • Fix Version/s: 4.7.1, 4.8, 6.0
    • Component/s: search
    • Labels:
      None

      Description

      SOLR-5354 added unmarshalling of distributed sort field values in QueryComponent.mergeIds(), but incorrectly performs this (unmarshalling all docs' sort field values) for every doc, and stores the result with each doc. This is unnecessary, inefficient, and extremely wasteful of memory.

      In an offline conversation, Alexey Serba described the issue to me and located the likely problem, and Chris Hostetter (Unused) located the problem code via inspection.

      This bug is very likely the problem described on the solr-user mailing list here: SolrCloud constantly crashes after upgrading to Solr 4.7

        Issue Links

          Activity

          Hide
          Steve Rowe added a comment -

          Simple patch with fix.

          Alexey Serba has confirmed that this solved the excessive memory use issue he saw.

          Committing shortly.

          Show
          Steve Rowe added a comment - Simple patch with fix. Alexey Serba has confirmed that this solved the excessive memory use issue he saw. Committing shortly.
          Hide
          ASF subversion and git services added a comment -

          Commit 1578434 from Steve Rowe in branch 'dev/trunk'
          [ https://svn.apache.org/r1578434 ]

          SOLR-5875: QueryComponent.mergeIds() unmarshals all docs' sort field values once per doc instead of once per shard.

          Show
          ASF subversion and git services added a comment - Commit 1578434 from Steve Rowe in branch 'dev/trunk' [ https://svn.apache.org/r1578434 ] SOLR-5875 : QueryComponent.mergeIds() unmarshals all docs' sort field values once per doc instead of once per shard.
          Hide
          ASF subversion and git services added a comment -

          Commit 1578435 from Steve Rowe in branch 'dev/branches/branch_4x'
          [ https://svn.apache.org/r1578435 ]

          SOLR-5875: QueryComponent.mergeIds() unmarshals all docs' sort field values once per doc instead of once per shard. (merged trunk r1578434)

          Show
          ASF subversion and git services added a comment - Commit 1578435 from Steve Rowe in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1578435 ] SOLR-5875 : QueryComponent.mergeIds() unmarshals all docs' sort field values once per doc instead of once per shard. (merged trunk r1578434)
          Hide
          ASF subversion and git services added a comment -

          Commit 1578438 from Steve Rowe in branch 'dev/branches/lucene_solr_4_7'
          [ https://svn.apache.org/r1578438 ]

          SOLR-5875: QueryComponent.mergeIds() unmarshals all docs' sort field values once per doc instead of once per shard. (merged trunk r1578434)

          Show
          ASF subversion and git services added a comment - Commit 1578438 from Steve Rowe in branch 'dev/branches/lucene_solr_4_7' [ https://svn.apache.org/r1578438 ] SOLR-5875 : QueryComponent.mergeIds() unmarshals all docs' sort field values once per doc instead of once per shard. (merged trunk r1578434)
          Hide
          Steve Rowe added a comment -

          Committed to trunk, branch_4x and the lucene_solr_4_7 branch.

          Show
          Steve Rowe added a comment - Committed to trunk, branch_4x and the lucene_solr_4_7 branch.
          Hide
          Erick Erickson added a comment -

          Hmmm, is it possible to have the original person who posted the problem give it a test run? For something like this it'd be good to have some proof that if fixes the problem described.

          Just a thought.

          Erick

          Show
          Erick Erickson added a comment - Hmmm, is it possible to have the original person who posted the problem give it a test run? For something like this it'd be good to have some proof that if fixes the problem described. Just a thought. Erick
          Hide
          Steve Rowe added a comment -

          Erick, as I mentioned above, Alexey Serba gave it a test run and it fixed the memory issue he saw.

          Show
          Steve Rowe added a comment - Erick, as I mentioned above, Alexey Serba gave it a test run and it fixed the memory issue he saw.
          Hide
          Steve Rowe added a comment -

          Bulk close 4.7.1 issues

          Show
          Steve Rowe added a comment - Bulk close 4.7.1 issues

            People

            • Assignee:
              Steve Rowe
              Reporter:
              Steve Rowe
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development