Lucene - Core
  1. Lucene - Core
  2. LUCENE-6049

Cryptic exception if all docs in a segment hit non-aborting exceptions before adding their doc values

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.10.3, 5.0, 6.0
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I hit this while working on LUCENE-6005:

      If you add a document with a single field that's both indexed and has doc values, and during inversion it hits a non-aborting exception, and all docs for a given segment had this happen, then you'll hit this confusing exception:

      java.lang.AssertionError: segment=_0(6.0.0):C2: field="test" has docValues but did not write them
      	at __randomizedtesting.SeedInfo.seed([21BFA52E65A19C81:3A824781C0F77629]:0)
      	at org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:146)
      	at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:93)
      	at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:440)
      	at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:511)
      	at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:622)
      	at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3016)
      	at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:2992)
      	at org.apache.lucene.index.IndexWriter.shutdown(IndexWriter.java:946)
      	at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:991)
      	at org.apache.lucene.index.TestDocValuesIndexing.testExcIndexingDocBeforeDocValues(TestDocValuesIndexing.java:927)
      

      The good news here is that exception is new from LUCENE-6019 and it prevents this case from causing index corruption, but the bad news is, you shouldn't even get an exception writing the segment in the first place.

      1. LUCENE-6049.patch
        31 kB
        Michael McCandless
      2. LUCENE-6049.patch
        4 kB
        Michael McCandless

        Activity

        Hide
        Michael McCandless added a comment -

        Patch w/ test and fix.

        I think we have to move away from this dangerous FieldInfos.addOrUpdate(String,IndexableFieldType) method, but this is just a baby step (fixing it to not carry over DocValuesType).

        Show
        Michael McCandless added a comment - Patch w/ test and fix. I think we have to move away from this dangerous FieldInfos.addOrUpdate(String,IndexableFieldType) method, but this is just a baby step (fixing it to not carry over DocValuesType).
        Hide
        Michael McCandless added a comment -

        Actually I'm going to try to nuke FieldInfos.addOrUpdate entirely: this is a scary method and it has caused too many bugs over the years.

        Show
        Michael McCandless added a comment - Actually I'm going to try to nuke FieldInfos.addOrUpdate entirely: this is a scary method and it has caused too many bugs over the years.
        Hide
        Michael McCandless added a comment -

        OK I managed to nuke the horrible FieldInfos.addOrUpdate(String,IndexableFieldType) method.

        I also renamed docValueType -> docValuesType; I'll leave that out when
        I backport to 4.10.3. In fact for 4.10.x I think I'll just do my
        original (less risky) patch, just removing updating of DocValuesType
        from FieldInfos.addOrUpdate.

        Show
        Michael McCandless added a comment - OK I managed to nuke the horrible FieldInfos.addOrUpdate(String,IndexableFieldType) method. I also renamed docValueType -> docValuesType; I'll leave that out when I backport to 4.10.3. In fact for 4.10.x I think I'll just do my original (less risky) patch, just removing updating of DocValuesType from FieldInfos.addOrUpdate.
        Hide
        ASF subversion and git services added a comment -

        Commit 1637524 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1637524 ]

        LUCENE-6049: don't throw exc writing segment if document hit non-aborting exc

        Show
        ASF subversion and git services added a comment - Commit 1637524 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1637524 ] LUCENE-6049 : don't throw exc writing segment if document hit non-aborting exc
        Hide
        ASF subversion and git services added a comment -

        Commit 1637528 from Michael McCandless in branch 'dev/branches/branch_5x'
        [ https://svn.apache.org/r1637528 ]

        LUCENE-6049: don't throw exc writing segment if document hit non-aborting exc

        Show
        ASF subversion and git services added a comment - Commit 1637528 from Michael McCandless in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1637528 ] LUCENE-6049 : don't throw exc writing segment if document hit non-aborting exc
        Hide
        ASF subversion and git services added a comment -

        Commit 1637529 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1637529 ]

        LUCENE-6049: changes entry

        Show
        ASF subversion and git services added a comment - Commit 1637529 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1637529 ] LUCENE-6049 : changes entry
        Hide
        ASF subversion and git services added a comment -

        Commit 1637533 from Michael McCandless in branch 'dev/branches/lucene_solr_4_10'
        [ https://svn.apache.org/r1637533 ]

        LUCENE-6049: don't throw exc writing segment if document hit non-aborting exc

        Show
        ASF subversion and git services added a comment - Commit 1637533 from Michael McCandless in branch 'dev/branches/lucene_solr_4_10' [ https://svn.apache.org/r1637533 ] LUCENE-6049 : don't throw exc writing segment if document hit non-aborting exc
        Hide
        Anshum Gupta added a comment -

        Bulk close after 5.0 release.

        Show
        Anshum Gupta added a comment - Bulk close after 5.0 release.

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development