Uploaded image for project: 'Lucene - Core'
  1. Lucene - Core
  2. LUCENE-880

DocumentWriter closes TokenStreams too early

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      The DocumentWriter closes a TokenStream as soon as it has consumed its tokens. The javadoc of TokenStream.close() says that it releases resources associated with the stream. However, the DocumentWriter keeps references of the resources (i. e. payload byte arrays, term strings) until it writes the postings to the new segment, which means that DocumentWriter should call TokenStream.close() after it has written the postings.

      This problem occurs in multithreaded applications where e. g. pooling is used for the resources. My patch adds a new test to TestPayloads which shows this problem. Multiple threads add documents with payloads to an index and use a pool of byte arrays for the payloads. TokenStream.close() puts the byte arrays back into the pool. The test fails with the old version but runs successfully with the patched version.

      All other units tests pass as well.

      1. lucene-880.patch
        11 kB
        Michael Busch

        Activity

        Hide
        michaelbusch Michael Busch added a comment -

        Patch with new unit test.

        Show
        michaelbusch Michael Busch added a comment - Patch with new unit test.
        Hide
        michaelbusch Michael Busch added a comment -

        Committed.

        Show
        michaelbusch Michael Busch added a comment - Committed.

          People

          • Assignee:
            michaelbusch Michael Busch
            Reporter:
            michaelbusch Michael Busch
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development