Lucene - Core
  1. Lucene - Core
  2. LUCENE-5228

IndexWriter.addIndexes copies raw files but acquires no locks

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 4.7, 6.0
    • Component/s: None
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      I see stuff like: "merge problem with lucene 3 and 4 indices" (from solr users list), and cannot even think how to respond to these users because so many things can go wrong with IndexWriter.addIndexes(Directory)

      it currently has in its javadocs:

      NOTE: the index in each Directory must not be changed (opened by a writer) while this method is running. This method does not acquire a write lock in each input Directory, so it is up to the caller to enforce this.

      This method should be acquiring locks: its copying RAW FILES. Otherwise we should remove it. If someone doesnt like that, or is mad because its 10ns slower, they can use NoLockFactory.

      1. LUCENE-5228.patch
        15 kB
        Michael McCandless

        Activity

        Hide
        Shai Erera added a comment -

        The problem is with Directories that don't support locking, e.g. on HDFS. But I guess NoLockFactory is a reasonable solution for them. I don't think there's any performance concern here because addIndexes(Directory...) is doing so much work (depending on the index size of course), that acquiring a lock on each Directory seems negligible.

        Let's do that? And also change the jdoc so explicitly state that and the NoLockFactory solution for Directories that cannot support locking?

        Show
        Shai Erera added a comment - The problem is with Directories that don't support locking, e.g. on HDFS. But I guess NoLockFactory is a reasonable solution for them. I don't think there's any performance concern here because addIndexes(Directory...) is doing so much work (depending on the index size of course), that acquiring a lock on each Directory seems negligible. Let's do that? And also change the jdoc so explicitly state that and the NoLockFactory solution for Directories that cannot support locking?
        Hide
        Michael McCandless added a comment -

        How about something like this?

        I fixed Lock to implement Closeable, and just added the acquire/close inside IW.addIndexes(Dir[]).

        Show
        Michael McCandless added a comment - How about something like this? I fixed Lock to implement Closeable, and just added the acquire/close inside IW.addIndexes(Dir[]).
        Hide
        Robert Muir added a comment -

        Very simple approach! I like it.

        Show
        Robert Muir added a comment - Very simple approach! I like it.
        Hide
        ASF subversion and git services added a comment -

        Commit 1561404 from Michael McCandless in branch 'dev/trunk'
        [ https://svn.apache.org/r1561404 ]

        LUCENE-5228: IndexWriter.addIndexes(Directory[]) now acquires the IW write lock on the incoming indices to ensure there are no active IndexWriters in those directories

        Show
        ASF subversion and git services added a comment - Commit 1561404 from Michael McCandless in branch 'dev/trunk' [ https://svn.apache.org/r1561404 ] LUCENE-5228 : IndexWriter.addIndexes(Directory[]) now acquires the IW write lock on the incoming indices to ensure there are no active IndexWriters in those directories
        Hide
        ASF subversion and git services added a comment -

        Commit 1561412 from Michael McCandless in branch 'dev/branches/branch_4x'
        [ https://svn.apache.org/r1561412 ]

        LUCENE-5228: IndexWriter.addIndexes(Directory[]) now acquires the IW write lock on the incoming indices to ensure there are no active IndexWriters in those directories

        Show
        ASF subversion and git services added a comment - Commit 1561412 from Michael McCandless in branch 'dev/branches/branch_4x' [ https://svn.apache.org/r1561412 ] LUCENE-5228 : IndexWriter.addIndexes(Directory[]) now acquires the IW write lock on the incoming indices to ensure there are no active IndexWriters in those directories
        Hide
        Michael McCandless added a comment -

        Thanks Rob, and Littlestar!

        Show
        Michael McCandless added a comment - Thanks Rob, and Littlestar!
        Hide
        Littlestar added a comment - - edited

        not fixed in 4.6.1? Thanks.

        >>>I see stuff like: "merge problem with lucene 3 and 4 indices" (from solr users list), and cannot even think how to respond to these users because so many things can go wrong with IndexWriter.addIndexes(Directory)
        may related to LUCENE-5377

        Show
        Littlestar added a comment - - edited not fixed in 4.6.1? Thanks. >>>I see stuff like: "merge problem with lucene 3 and 4 indices" (from solr users list), and cannot even think how to respond to these users because so many things can go wrong with IndexWriter.addIndexes(Directory) may related to LUCENE-5377
        Hide
        Littlestar added a comment -

        >>>IndexWriter.addIndexes(Directory[]) now acquires the IW write lock
        I think the write lock is too expensive for my app. a SnapshotDeletionPolicy is OK for some situation.
        is there a way equal to IndexWriter.addIndexes(IndexCommit)?

        Show
        Littlestar added a comment - >>>IndexWriter.addIndexes(Directory[]) now acquires the IW write lock I think the write lock is too expensive for my app. a SnapshotDeletionPolicy is OK for some situation. is there a way equal to IndexWriter.addIndexes(IndexCommit)?

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Robert Muir
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development