Solr
  1. Solr
  2. SOLR-5418

Background merge after field removed from solr.xml causes error

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 4.5
    • Fix Version/s: 4.6, Trunk
    • Component/s: Schema and Analysis
    • Labels:
      None

      Description

      Problem from the user's list, cut/pasted below. Robert Muir hacked out a quick patch he pasted on the dev list, I'll append it shortly.

      I am working at implementing solr to work as the search backend for our web
      system. So far things have been going well, but today I made some schema
      changes and now things have broken.

      I updated the schema.xml file and reloaded the core (via the admin
      interface). No errors were reported in the logs.

      I then pushed 100 records to be indexed. A call to Commit afterwards
      seemed fine, however my next call for Optimize caused the following errors:

      java.io.IOException: background merge hit exception:
      _2n(4.4):C4263/154 _30(4.4):C134 _32(4.4):C10 _31(4.4):C10 into _37
      [maxNumSegments=1]

      null:java.io.IOException: background merge hit exception:
      _2n(4.4):C4263/154 _30(4.4):C134 _32(4.4):C10 _31(4.4):C10 into _37
      [maxNumSegments=1]

      Unfortunately, googling for background merge hit exception came up
      with 2 thing: a corrupt index or not enough free space. The host
      machine that's hosting solr has 227 out of 229GB free (according to df
      -h), so that's not it.

      I then ran CheckIndex on the index, and got the following results:
      http://apaste.info/gmGU

      As someone who is new to solr and lucene, as far as I can tell this
      means my index is fine. So I am coming up at a loss. I'm somewhat sure
      that I could probably delete my data directory and rebuild it but I am
      more interested in finding out why is it having issues, what is the
      best way to fix it, and what is the best way to prevent it from
      happening when this goes into production.

      Does anyone have any advice that may help?

      I helped Matthew find the logs and he posted this stack trace:

      1691103929 [http-bio-8080-exec-3] INFO org.apache.solr.update.UpdateHandler â start commit

      {,optimize=true,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}

      1691104153 [http-bio-8080-exec-3] INFO org.apache.solr.update.processor.LogUpdateProcessor â [dbqItems] webapp=/solr path=/update params=

      {optimize=true&_=1382999386564&wt=json&waitFlush=true}

      {} 0 224
      1691104154 [http-bio-8080-exec-3] ERROR org.apache.solr.core.SolrCore â java.io.IOException: background merge hit exception: _2n(4.4):C4263/154 _30(4.4):C134 _32(4.4):C10 _31(4.4):C10 into _39 [maxNumSegments=1]
      at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1714)
      at org.apache.lucene.index.IndexWriter.forceMerge(IndexWriter.java:1650)
      at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:530)
      at org.apache.solr.update.processor.RunUpdateProcessor.processCommit(RunUpdateProcessorFactory.java:95)
      at org.apache.solr.update.processor.UpdateRequestProcessor.processCommit(UpdateRequestProcessor.java:64)
      at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalCommit(DistributedUpdateProcessor.java:1240)
      at org.apache.solr.update.processor.DistributedUpdateProcessor.processCommit(DistributedUpdateProcessor.java:1219)
      at org.apache.solr.update.processor.LogUpdateProcessor.processCommit(LogUpdateProcessorFactory.java:157)
      at org.apache.solr.handler.RequestHandlerUtils.handleCommit(RequestHandlerUtils.java:69)
      at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:68)
      at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
      at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
      at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:659)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)
      at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
      at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
      at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
      at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
      at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
      at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
      at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
      at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
      at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
      at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
      at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1023)
      at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
      at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:679)
      Caused by: java.lang.IllegalArgumentException: no such field what
      at org.apache.solr.core.SchemaCodecFactory$1.getPostingsFormatForField(SchemaCodecFactory.java:59)
      at org.apache.lucene.codecs.lucene42.Lucene42Codec$1.getPostingsFormatForField(Lucene42Codec.java:59)
      at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.addField(PerFieldPostingsFormat.java:102)
      at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:71)
      at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:365)
      at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:98)
      at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:3772)
      at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3376)
      at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
      at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)

      1. SOLR-5418.patch
        3 kB
        Erick Erickson
      2. SOLR-5418.patch
        3 kB
        Robert Muir
      3. SOLR-5418.patch
        10 kB
        Erick Erickson
      4. SOLR-5418.patch
        11 kB
        Erick Erickson

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        2d 13h 14m 1 Erick Erickson 05/Nov/13 01:54
        Erick Erickson made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 4.6 [ 12325000 ]
        Fix Version/s 5.0 [ 12321664 ]
        Resolution Fixed [ 1 ]
        Hide
        Erick Erickson added a comment -

        Thanks Matthew and Robert!

        Show
        Erick Erickson added a comment - Thanks Matthew and Robert!
        Hide
        ASF subversion and git services added a comment -

        Commit 1538848 from Erick Erickson in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1538848 ]

        Fix for SOLR-5418, merge after altering schema produces error

        Show
        ASF subversion and git services added a comment - Commit 1538848 from Erick Erickson in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1538848 ] Fix for SOLR-5418 , merge after altering schema produces error
        Hide
        ASF subversion and git services added a comment -

        Commit 1538802 from Erick Erickson in branch 'dev/trunk'
        [ https://svn.apache.org/r1538802 ]

        Fix for SOLR-5418, merge after altering schema produces error

        Show
        ASF subversion and git services added a comment - Commit 1538802 from Erick Erickson in branch 'dev/trunk' [ https://svn.apache.org/r1538802 ] Fix for SOLR-5418 , merge after altering schema produces error
        Erick Erickson made changes -
        Attachment SOLR-5418.patch [ 12612053 ]
        Hide
        Erick Erickson added a comment -

        Put entry in CHANGES.txt

        Show
        Erick Erickson added a comment - Put entry in CHANGES.txt
        Erick Erickson made changes -
        Attachment SOLR-5418.patch [ 12612052 ]
        Hide
        Erick Erickson added a comment -

        Patch back with a test.

        Show
        Erick Erickson added a comment - Patch back with a test.
        Hide
        Erick Erickson added a comment -

        Apparently, NONE of our test code hits the code in question. I notice that this is a 4.6 codec, am I missing something obvious here? Well of course I am. I just wish I knew what it was.

        Show
        Erick Erickson added a comment - Apparently, NONE of our test code hits the code in question. I notice that this is a 4.6 codec, am I missing something obvious here? Well of course I am. I just wish I knew what it was.
        Hide
        Erick Erickson added a comment -

        Well, this is annoying. I can generate this failure easily enough with running stock Solr (trunk or 4x), but I can't seem to get a test case to fail inside or outside of IntelliJ.

        I've tracked creating two separate segments (forced disk index). They are being merged into one. But in the test case, it's not getting to the code that Robert changed. I've verified that I am successfully changing the schema (tried to index a second doc with the removed field and it fails in the IDE).

        So I can easily enough reproduce the problem, but I'm having difficulty making a test case. I want a verified failure before I apply the patch...

        I'm in the midst of seeing if our test suite ever hits the code in question (Siigggh. Back to debugging by println).

        Any pointers gratefully received.

        Erick

        Show
        Erick Erickson added a comment - Well, this is annoying. I can generate this failure easily enough with running stock Solr (trunk or 4x), but I can't seem to get a test case to fail inside or outside of IntelliJ. I've tracked creating two separate segments (forced disk index). They are being merged into one. But in the test case, it's not getting to the code that Robert changed. I've verified that I am successfully changing the schema (tried to index a second doc with the removed field and it fails in the IDE). So I can easily enough reproduce the problem, but I'm having difficulty making a test case. I want a verified failure before I apply the patch... I'm in the midst of seeing if our test suite ever hits the code in question (Siigggh. Back to debugging by println). Any pointers gratefully received. Erick
        Hide
        Erick Erickson added a comment -

        Thanks Robert! Mostly I was making sure I didn't lose anything....

        Show
        Erick Erickson added a comment - Thanks Robert! Mostly I was making sure I didn't lose anything....
        Robert Muir made changes -
        Attachment SOLR-5418.patch [ 12611776 ]
        Hide
        Robert Muir added a comment -

        Here is the patch from my svn checkout.

        I sent it to the list really quick just to get an opinion on it. I don't remember why the current checks were added. I guess I can see a line of reasoning that its good for the user to know they are dragging unused shit around in their index.

        And I can agree with that, but I don't think its the job of the codec to fail a background merge to communicate such a thing to the user.

        Such a warning/failure could be implemented in other ways, for example a warmer in the example for "firstSearcher" event that looks at fieldinfos and warns or fails "YOU ARE DRAGGING AROUND BOGUS STUFF IN YOUR INDEX" if it finds things that don't match to the schema or something like that: and it would be easy for users to enable/disable.

        Show
        Robert Muir added a comment - Here is the patch from my svn checkout. I sent it to the list really quick just to get an opinion on it. I don't remember why the current checks were added. I guess I can see a line of reasoning that its good for the user to know they are dragging unused shit around in their index. And I can agree with that, but I don't think its the job of the codec to fail a background merge to communicate such a thing to the user. Such a warning/failure could be implemented in other ways, for example a warmer in the example for "firstSearcher" event that looks at fieldinfos and warns or fails "YOU ARE DRAGGING AROUND BOGUS STUFF IN YOUR INDEX" if it finds things that don't match to the schema or something like that: and it would be easy for users to enable/disable.
        Erick Erickson made changes -
        Assignee Erick Erickson [ erickerickson ]
        Erick Erickson made changes -
        Field Original Value New Value
        Attachment SOLR-5418.patch [ 12611760 ]
        Hide
        Erick Erickson added a comment -

        Patch constructed from cut-n-paste of Robert's quick code change on the dev list.

        Warning, it's cut/paste, not generated from svn so be warned....

        Show
        Erick Erickson added a comment - Patch constructed from cut-n-paste of Robert's quick code change on the dev list. Warning, it's cut/paste, not generated from svn so be warned....
        Erick Erickson created issue -

          People

          • Assignee:
            Erick Erickson
            Reporter:
            Erick Erickson
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development