Uploaded image for project: 'Solr'
  1. Solr
  2. SOLR-14254

Index backcompat break between 8.3.1 and 8.4.1

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      I believe I found a backcompat break between 8.4.1 and 8.3.1.

      I encountered this when a Solr 8.3.1 cluster was upgraded to 8.4.1. On 8.4. nodes, several collections had cores fail to come up with CorruptIndexException:

      2020-02-10 20:58:26.136 ERROR (coreContainerWorkExecutor-2-thread-1-processing-n:192.168.1.194:8983_solr) [   ] o.a.s.c.CoreContainer Error waiting for SolrCore to be loaded on startup => org.apache.sol
      r.common.SolrException: Unable to create core [testbackcompat_shard1_replica_n1]
              at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1313)
      org.apache.solr.common.SolrException: Unable to create core [testbackcompat_shard1_replica_n1]
              at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1313) ~[?:?]
              at org.apache.solr.core.CoreContainer.lambda$load$13(CoreContainer.java:788) ~[?:?]
              at com.codahale.metrics.InstrumentedExecutorService$InstrumentedCallable.call(InstrumentedExecutorService.java:202) ~[metrics-core-4.0.5.jar:4.0.5]
              at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
              at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:210) ~[?:?]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
              at java.lang.Thread.run(Thread.java:834) [?:?]
      Caused by: org.apache.solr.common.SolrException: Error opening new searcher
              at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1072) ~[?:?]
              at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
              at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292) ~[?:?]
              ... 7 more
      Caused by: org.apache.solr.common.SolrException: Error opening new searcher
              at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2182) ~[?:?]
              at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2302) ~[?:?]
              at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1132) ~[?:?]
              at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1013) ~[?:?]
              at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
              at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292) ~[?:?]
              ... 7 more
      Caused by: org.apache.lucene.index.CorruptIndexException: codec mismatch: actual codec=Lucene50PostingsWriterDoc vs expected codec=Lucene84PostingsWriterDoc (resource=MMapIndexInput(path="/Users/jasongerlowski/run/solrdata/data/testbackcompat_shard1_replica_n1/data/index/_0_FST50_0.doc"))
              at org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:208) ~[?:?]
              at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:198) ~[?:?]
              at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255) ~[?:?]
              at org.apache.lucene.codecs.lucene84.Lucene84PostingsReader.<init>(Lucene84PostingsReader.java:82) ~[?:?]
              at org.apache.lucene.codecs.memory.FSTPostingsFormat.fieldsProducer(FSTPostingsFormat.java:66) ~[?:?]
              at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.<init>(PerFieldPostingsFormat.java:315) ~[?:?]
              at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:395) ~[?:?]
              at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:114) ~[?:?]
              at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:84) ~[?:?]
              at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:177) ~[?:?]
              at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:219) ~[?:?]
              at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:109) ~[?:?]
              at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:526) ~[?:?]
              at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:116) ~[?:?]
              at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:92) ~[?:?]
              at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:39) ~[?:?]
              at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2146) ~[?:?]
              at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2302) ~[?:?]
              at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1132) ~[?:?]
              at org.apache.solr.core.SolrCore.<init>(SolrCore.java:1013) ~[?:?]
              at org.apache.solr.core.SolrCore.<init>(SolrCore.java:901) ~[?:?]
              at org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1292) ~[?:?]
              ... 7 more
      

      Only some cores were affected. After some digging, we noticed that these cores were unique in that they had a field set up to use Solr's Text Tagger. Not the use of a non-default postings-format (FST50).

              <fieldType class="solr.TextField" name="tagger" omitNorms="true" omitTermFreqAndPositions="true" postingsFormat="FST50">
                <analyzer type="index">
                  <tokenizer class="solr.StandardTokenizerFactory"/>
                  <filter class="solr.EnglishPossessiveFilterFactory"/>
                  <filter class="solr.ASCIIFoldingFilterFactory"/>
                  <filter class="solr.LowerCaseFilterFactory"/>
                  <filter class="solr.ConcatenateGraphFilterFactory" preservePositionIncrements="false"/>
                </analyzer>
                <analyzer type="query">
                  <tokenizer class="solr.StandardTokenizerFactory"/>
                  <filter class="solr.EnglishPossessiveFilterFactory"/>
                  <filter class="solr.ASCIIFoldingFilterFactory"/>
                  <filter class="solr.LowerCaseFilterFactory"/>
                </analyzer>
              </fieldType>
              <field name="tagger_field" type="tagger" indexed="true" stored="true" />
      

      I've since been able to reproduce the problem locally with the following steps:

      1. Start Solr 8.3.1 with a data directory /some/data/dir
      2. Modify the default configset (or create a new configset) by adding the fieldType/field above to your schema.
      3. Index a document that contains a "tagger_field" value and commit.
      4. Stop Solr 8.3.1
      5. Start Solr 8.4.1 pointed at the /some/data/dir as its data directory.

      At this point, the core should be down and you should see the CorruptIndexException in your logs.

      This seems related to LUCENE-9027, though I don't know enough about how Lucene handles ensuring support for older indices to know exactly what happened there.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                gerlowskija Jason Gerlowski
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h 10m
                  1h 10m