Solr
  1. Solr
  2. SOLR-4370

Ability to control the commit behaviour of commitWithin

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.0, 4.1
    • Fix Version/s: 4.2, 6.0
    • Component/s: replication (java)
    • Labels:
      None

      Description

      We need the ability to control the hard/soft commit behaviour of commitWithin parameter.

      Since Solr 4.0, the commitWithin's performs a soft-commit which prevents slaves from picking up the changes in a master/slave configuration.

      The behaviour I'm thinking is as follows:

      1. By default, commitWithin will trigger soft commits.

      2. But this behaviour can be overwritten in solrconfig.xml to allow commitWithin to perform hard commits, which will allow slaves to pick up the changes.

      <commitWithin>
      <forceHardCommit>true</forceHardCommit> <!-- default is false -->
      </commitWithin>

      Related to SOLR-4100

      1. SOLR-4370.patch
        11 kB
        Mark Miller
      2. with_commit.txt
        3 kB
        Senthuran Sivananthan
      3. without_commit.txt
        3 kB
        Senthuran Sivananthan

        Issue Links

          Activity

          Hide
          Mark Miller added a comment -

          Patch attached.

          Show
          Mark Miller added a comment - Patch attached.
          Hide
          Senthuran Sivananthan added a comment - - edited

          Tested the patch on 4.2 from http://svn.apache.org/repos/asf/lucene/dev/branches/branch_4x/ and found a couple issues when solr.commitwithin.softcommit = false. The setting allows the slaves to replicate the indices and I can see the index files updated after commitWithin timeframe.

          The issues are:

          1/ Stats on the collection overview page doesn't get updated. This includes the generation & version info. An explicit commit (i.e. http://localhost:8983/solr/collection1/update?commit=true) will update the stats.

          2/ Can't find the indexed docs on the master. An explicit commit solves the issue.

          I think the patch solves the original request but introduced a couple of side-effects.


          Additional info
          ---------------

          I did a quick diff between explicit commit (commit=true) vs. commitWithin=2000 vs. HardAutoCommitTest and here's what I found. May be I'm off track, but thought I share:

          1. commit = true

          • A new searcher is registered

          Jan 28, 2013 5:30:39 PM org.apache.solr.search.SolrIndexSearcher <init>
          INFO: Opening Searcher@47955387 main
          Jan 28, 2013 5:30:39 PM org.apache.solr.core.CachingDirectoryFactory close
          INFO: Releasing directory:\Tools\Solr\dev\branch_4x\solr\example\solr\collection1\data\index
          Jan 28, 2013 5:30:39 PM org.apache.solr.core.QuerySenderListener newSearcher
          INFO: QuerySenderListener sending requests to Searcher@47955387 main

          {StandardDirectoryReader(segments_4:9:nrt _0(4.2):C2 _2(4.2):C3)}

          Jan 28, 2013 5:30:39 PM org.apache.solr.update.DirectUpdateHandler2 commit
          INFO: end_commit_flush
          Jan 28, 2013 5:30:39 PM org.apache.solr.core.QuerySenderListener newSearcher
          INFO: QuerySenderListener done.
          Jan 28, 2013 5:30:39 PM org.apache.solr.core.SolrCore registerSearcher
          INFO: [collection1] Registered new searcher Searcher@47955387 main

          {StandardDirectoryReader(segments_4:9:nrt _0(4.2):C2 _2(4.2):C3)}

          2. commitWithin=N

          New search isn't registered

          Jan 28, 2013 5:31:47 PM org.apache.solr.search.SolrIndexSearcher <init>
          INFO: Opening Searcher@4d000e49 realtime
          Jan 28, 2013 5:31:47 PM org.apache.solr.update.DirectUpdateHandler2 commit
          INFO: end_commit_flush

          3. HardAutoCommitTest

          23619 T18 oasc.CachingDirectoryFactory.close Releasing directory:C:\Users\SENTHU~1\AppData\Local\Temp\solrtest-HardAutoCommitTest-1359411964946
          23640 T18 oass.SolrIndexSearcher.<init> Opening Searcher@3b0d579 main
          23642 T18 oasu.DirectUpdateHandler2.commit end_commit_flush
          23654 T17 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3b0d579 main

          {StandardDirectoryReader(segments_2:4:nrt _0(4.2):C1)}

          23655 T17 oasc.CachingDirectoryFactory.close Releasing directory:C:\Users\SENTHU~1\AppData\Local\Temp\solrtest-HardAutoCommitTest-1359411964946\index
          29234 T15 C1 REQ [collection1] webapp=null path=null params=

          {start=0&q=id:529&qt=standard&rows=20&version=2.2}

          hits=1 status=0 QTime=4

          Any way to register a new searcher?

          Show
          Senthuran Sivananthan added a comment - - edited Tested the patch on 4.2 from http://svn.apache.org/repos/asf/lucene/dev/branches/branch_4x/ and found a couple issues when solr.commitwithin.softcommit = false. The setting allows the slaves to replicate the indices and I can see the index files updated after commitWithin timeframe. The issues are: 1/ Stats on the collection overview page doesn't get updated. This includes the generation & version info. An explicit commit (i.e. http://localhost:8983/solr/collection1/update?commit=true ) will update the stats. 2/ Can't find the indexed docs on the master. An explicit commit solves the issue. I think the patch solves the original request but introduced a couple of side-effects. Additional info --------------- I did a quick diff between explicit commit (commit=true) vs. commitWithin=2000 vs. HardAutoCommitTest and here's what I found. May be I'm off track, but thought I share: 1. commit = true A new searcher is registered Jan 28, 2013 5:30:39 PM org.apache.solr.search.SolrIndexSearcher <init> INFO: Opening Searcher@47955387 main Jan 28, 2013 5:30:39 PM org.apache.solr.core.CachingDirectoryFactory close INFO: Releasing directory :\Tools\Solr\dev\branch_4x\solr\example\solr\collection1\data\index Jan 28, 2013 5:30:39 PM org.apache.solr.core.QuerySenderListener newSearcher INFO: QuerySenderListener sending requests to Searcher@47955387 main {StandardDirectoryReader(segments_4:9:nrt _0(4.2):C2 _2(4.2):C3)} Jan 28, 2013 5:30:39 PM org.apache.solr.update.DirectUpdateHandler2 commit INFO: end_commit_flush Jan 28, 2013 5:30:39 PM org.apache.solr.core.QuerySenderListener newSearcher INFO: QuerySenderListener done. Jan 28, 2013 5:30:39 PM org.apache.solr.core.SolrCore registerSearcher INFO: [collection1] Registered new searcher Searcher@47955387 main {StandardDirectoryReader(segments_4:9:nrt _0(4.2):C2 _2(4.2):C3)} 2. commitWithin=N New search isn't registered Jan 28, 2013 5:31:47 PM org.apache.solr.search.SolrIndexSearcher <init> INFO: Opening Searcher@4d000e49 realtime Jan 28, 2013 5:31:47 PM org.apache.solr.update.DirectUpdateHandler2 commit INFO: end_commit_flush 3. HardAutoCommitTest 23619 T18 oasc.CachingDirectoryFactory.close Releasing directory:C:\Users\SENTHU~1\AppData\Local\Temp\solrtest-HardAutoCommitTest-1359411964946 23640 T18 oass.SolrIndexSearcher.<init> Opening Searcher@3b0d579 main 23642 T18 oasu.DirectUpdateHandler2.commit end_commit_flush 23654 T17 oasc.SolrCore.registerSearcher [collection1] Registered new searcher Searcher@3b0d579 main {StandardDirectoryReader(segments_2:4:nrt _0(4.2):C1)} 23655 T17 oasc.CachingDirectoryFactory.close Releasing directory:C:\Users\SENTHU~1\AppData\Local\Temp\solrtest-HardAutoCommitTest-1359411964946\index 29234 T15 C1 REQ [collection1] webapp=null path=null params= {start=0&q=id:529&qt=standard&rows=20&version=2.2} hits=1 status=0 QTime=4 Any way to register a new searcher?
          Hide
          Mark Miller added a comment -

          Sounds like openSearcher is set to false. Can you pull the commit lines from your logs (that show the params)?

          Show
          Mark Miller added a comment - Sounds like openSearcher is set to false. Can you pull the commit lines from your logs (that show the params)?
          Hide
          Senthuran Sivananthan added a comment -

          Mark, I just updated my comment about (didn't see your post till now). Can you see if it makes sense or whether you need additional info?

          Show
          Senthuran Sivananthan added a comment - Mark, I just updated my comment about (didn't see your post till now). Can you see if it makes sense or whether you need additional info?
          Hide
          Senthuran Sivananthan added a comment - - edited

          Logs attached.

          Show
          Senthuran Sivananthan added a comment - - edited Logs attached.
          Hide
          Hoss Man added a comment -

          Before going too far down the rabit hole of how to name some of this stuff in configs, it might be worthwhle to consider some of the comments/concerns raised in SOLR-3539 about hte existing naming of softCommit and start taking baby steps towards clearer names for things.

          (ie: instead of "forecHardCommit" perhaps there should be a "persist" setting and an "openSearcher" setting)

          Show
          Hoss Man added a comment - Before going too far down the rabit hole of how to name some of this stuff in configs, it might be worthwhle to consider some of the comments/concerns raised in SOLR-3539 about hte existing naming of softCommit and start taking baby steps towards clearer names for things. (ie: instead of "forecHardCommit" perhaps there should be a "persist" setting and an "openSearcher" setting)
          Hide
          Senthuran Sivananthan added a comment -

          OK, I just saw what you meant by openSearcher=false

          I updated:

          <autoCommit>
          <maxTime>15000</maxTime>
          <openSearcher>false</openSearcher>
          </autoCommit>

          to

          <autoCommit>
          <maxTime>15000</maxTime>
          <openSearcher>true</openSearcher>
          </autoCommit>

          and that seem to do the trick. I'll test and let you know.

          Show
          Senthuran Sivananthan added a comment - OK, I just saw what you meant by openSearcher=false I updated: <autoCommit> <maxTime>15000</maxTime> <openSearcher>false</openSearcher> </autoCommit> to <autoCommit> <maxTime>15000</maxTime> <openSearcher>true</openSearcher> </autoCommit> and that seem to do the trick. I'll test and let you know.
          Hide
          Senthuran Sivananthan added a comment -

          The 2 issues that I saw are now resolved.

          I needed to set openSearcher = true (by default it was false in my config).

          Thanks for the quick turnaround Mark.

          Show
          Senthuran Sivananthan added a comment - The 2 issues that I saw are now resolved. I needed to set openSearcher = true (by default it was false in my config). Thanks for the quick turnaround Mark.
          Hide
          Mark Miller added a comment -

          Hmm, that is odd though. Seems rather unintuitive that auto commit settings would apply to commitWithin.

          Show
          Mark Miller added a comment - Hmm, that is odd though. Seems rather unintuitive that auto commit settings would apply to commitWithin.
          Hide
          Senthuran Sivananthan added a comment -

          At the same time, if we don't commit during update, isn't it the auto commit that's actually doing the commit based on maxTime, maxDoc or commitWithin?

          Show
          Senthuran Sivananthan added a comment - At the same time, if we don't commit during update, isn't it the auto commit that's actually doing the commit based on maxTime, maxDoc or commitWithin?
          Hide
          Mark Miller added a comment -

          Yes, the commitWithin and auto commit code is tied up as an impl detail, but not from a user perspective. You shouldn't have to know a thing about auto commit or have any config for it to work fully with commitWithin.

          Show
          Mark Miller added a comment - Yes, the commitWithin and auto commit code is tied up as an impl detail, but not from a user perspective. You shouldn't have to know a thing about auto commit or have any config for it to work fully with commitWithin.
          Hide
          Mark Miller added a comment -

          Alright - that's really a separate issue and it's bothering me less over time.

          I'll commit as is without further input.

          Show
          Mark Miller added a comment - Alright - that's really a separate issue and it's bothering me less over time. I'll commit as is without further input.
          Hide
          Commit Tag Bot added a comment -

          [trunk commit] Mark Robert Miller
          http://svn.apache.org/viewvc?view=revision&revision=1441483

          SOLR-4370: Allow configuring commitWithin to do hard commits.

          Show
          Commit Tag Bot added a comment - [trunk commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1441483 SOLR-4370 : Allow configuring commitWithin to do hard commits.
          Hide
          Commit Tag Bot added a comment -

          [branch_4x commit] Mark Robert Miller
          http://svn.apache.org/viewvc?view=revision&revision=1441490

          SOLR-4370: Allow configuring commitWithin to do hard commits.

          Show
          Commit Tag Bot added a comment - [branch_4x commit] Mark Robert Miller http://svn.apache.org/viewvc?view=revision&revision=1441490 SOLR-4370 : Allow configuring commitWithin to do hard commits.
          Hide
          Uwe Schindler added a comment -

          Closed after release.

          Show
          Uwe Schindler added a comment - Closed after release.

            People

            • Assignee:
              Mark Miller
              Reporter:
              Senthuran Sivananthan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development