Lucene - Core
  1. Lucene - Core
  2. LUCENE-2561

Fix exception handling and thread safety in realtime branch

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Realtime Branch
    • Component/s: core/index
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Several tests are currently failing in the realtime branch - most of them due to thread safety problems (often exceptions in ConcurrentMergeScheduler) and in tests that test for aborting and non-aborting exceptions.

      1. lucene-2561.patch
        57 kB
        Michael Busch

        Activity

        Hide
        Michael Busch added a comment -

        The patch fixes most of the threading and exception issues.

        Now >99% of the core tests pass! Some failures are expected, because some features aren't implemented yet (e.g. flush by RAM or maxBufferedDeletes). A test though that I still want to fix with this patch is TestStressIndexing2 - not sure yet what's going on.

        Other changes:

        • Factored ReaderPool out of IndexWriter into its own class
        • Added a FilterDirectory that forwards all method calls to a delegate
        • Use an extended FilterDirectory in DW to track all files the consumers and codecs open, so that they can be closed on abort
        • Fixed some more nocommits

        Using the FilterDirectory might not be the cleanest approach? Maybe a IndexOutputFactory or something would be cleaner? Or maybe on abort we should just delete all files that have the prefix of the segment(s) the DWPT(s) were working on? This should be possible now that the shared doc stores are gone and no files are shared anymore across segments.

        Show
        Michael Busch added a comment - The patch fixes most of the threading and exception issues. Now >99% of the core tests pass! Some failures are expected, because some features aren't implemented yet (e.g. flush by RAM or maxBufferedDeletes). A test though that I still want to fix with this patch is TestStressIndexing2 - not sure yet what's going on. Other changes: Factored ReaderPool out of IndexWriter into its own class Added a FilterDirectory that forwards all method calls to a delegate Use an extended FilterDirectory in DW to track all files the consumers and codecs open, so that they can be closed on abort Fixed some more nocommits Using the FilterDirectory might not be the cleanest approach? Maybe a IndexOutputFactory or something would be cleaner? Or maybe on abort we should just delete all files that have the prefix of the segment(s) the DWPT(s) were working on? This should be possible now that the shared doc stores are gone and no files are shared anymore across segments.
        Hide
        Michael Busch added a comment -

        Committed the current patch to the realtime branch. (revision 979856)

        Leaving this issue open to fix TestStressIndexing2 and for more cleanup.

        Show
        Michael Busch added a comment - Committed the current patch to the realtime branch. (revision 979856) Leaving this issue open to fix TestStressIndexing2 and for more cleanup.
        Hide
        Michael Busch added a comment -

        TestStressIndexing2 is not failing because of concurrency problems, so I'm closing this issue.
        All contrib tests pass now too!

        The reason why TestStressIndexing2 is failing is that deletes and sequenceIDs aren't fully implemented yet. The remapDeletes step is still commented out, which results in a wrong behavior as soon as segment merges happen while deletes are buffered. (I'll use LUCENE-2558 to fix that)

        Show
        Michael Busch added a comment - TestStressIndexing2 is not failing because of concurrency problems, so I'm closing this issue. All contrib tests pass now too! The reason why TestStressIndexing2 is failing is that deletes and sequenceIDs aren't fully implemented yet. The remapDeletes step is still commented out, which results in a wrong behavior as soon as segment merges happen while deletes are buffered. (I'll use LUCENE-2558 to fix that)

          People

          • Assignee:
            Michael Busch
            Reporter:
            Michael Busch
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development