Lucene - Core
  1. Lucene - Core
  2. LUCENE-771

Change default write lock file location to index directory (not java.io.tmpdir)

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.1
    • Fix Version/s: 2.1
    • Component/s: core/store
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      Now that readers are read-only, we no longer need to store lock files
      in a different global lock directory than the index directory. This
      has been a source of confusion and caused problems to users in the
      past.

      Furthermore, once the write lock is stored in the index directory, it
      no longer needs the big digest prefix that was previously required
      to make sure lock files in the global lock directory, from different
      indexes, did not conflict.

      This way, all files related to an index will appear in a single
      directory. And you can easily list that directory to see if a
      "write.lock" is present to check whether a writer is open on the
      index.

      Note that this change just affects how FSDirectory creates its default
      lockFactory if no lockFactory was specified. It is still possible
      (just no longer the default) to pick a different directory to store
      your lock files by pre-instantiating your own LockFactory.

      As part of this I would like to remove LOCK_DIR and the no-argument
      constructor, in SimpleFSLockFactory and NativeFSLockFactory. I don't
      think we should have the notion of a global default lock directory
      anymore. This is actually an API change. However, neither
      SimpleFSLockFactory nor NativeFSLockFactory haver been released yet,
      so I think this API removal is allowed?

      Finally I want to deprecate (but not yet remove, because this has been
      in the API for many releases) the static LOCK_DIR that's in
      FSDirectory. But it's now entirely unused.

      See here for discussion leading to this:

      http://www.gossamer-threads.com/lists/lucene/java-dev/43940

        Activity

        Hide
        Yonik Seeley added a comment -

        Sounds good, I agree with all the changes you outlined.

        Show
        Yonik Seeley added a comment - Sounds good, I agree with all the changes you outlined.
        Hide
        Michael McCandless added a comment -

        Thanks for the review Yonik!

        And thanks for pointing this out Marvin!

        I've committed this.

        Show
        Michael McCandless added a comment - Thanks for the review Yonik! And thanks for pointing this out Marvin! I've committed this.
        Hide
        Doron Cohen added a comment -

        I have a question on this change - though I didn't look at the code yet - we don't support backwards compatability with regard to existing lock files, right? Specifically, if there exist an older version index with locks in tmp dir, and the Lucene lib is replaced for one client at a time, while other clients are running, there would be a problem, possible corruption. So who ever is migrating from an index before this change to an index with this change should make sure that all clients are upgraded at the time. At least all clients that may attempt to write.

        Show
        Doron Cohen added a comment - I have a question on this change - though I didn't look at the code yet - we don't support backwards compatability with regard to existing lock files, right? Specifically, if there exist an older version index with locks in tmp dir, and the Lucene lib is replaced for one client at a time, while other clients are running, there would be a problem, possible corruption. So who ever is migrating from an index before this change to an index with this change should make sure that all clients are upgraded at the time. At least all clients that may attempt to write.
        Hide
        Michael McCandless added a comment -

        Yes, that is true.

        But there are also quite a few other changes in 2.1 (eg the file format changes) that would also generally necessitate that all places that will share the given index, upgrade to 2.1 at once.

        Show
        Michael McCandless added a comment - Yes, that is true. But there are also quite a few other changes in 2.1 (eg the file format changes) that would also generally necessitate that all places that will share the given index, upgrade to 2.1 at once.
        Hide
        Doron Cohen added a comment -

        Is that true? I thought that for previous format changes, the combination of -

        { (1) point-in-time index reading by readers (2) backwards compatibility (3) locks }

        - allowed not to require this.

        Show
        Doron Cohen added a comment - Is that true? I thought that for previous format changes, the combination of - { (1) point-in-time index reading by readers (2) backwards compatibility (3) locks } - allowed not to require this.
        Hide
        Michael McCandless added a comment -

        You're right, backwards compatibility will allow a 2.1 client to
        read/write to an older index, but as soon as a 2.1 client writes to
        the index then all pre-2.1 readers won't be able to read it. You're
        right that if these readers are not reopened then "point in time"
        would allow them to keep running pre-2.1.

        So I think you could individually update each reader to 2.1, but then
        (after readers are updated), all writers would need to go to 2.1 at
        once, to make sure they all "agree" that the write lock is now in the
        index.

        Show
        Michael McCandless added a comment - You're right, backwards compatibility will allow a 2.1 client to read/write to an older index, but as soon as a 2.1 client writes to the index then all pre-2.1 readers won't be able to read it. You're right that if these readers are not reopened then "point in time" would allow them to keep running pre-2.1. So I think you could individually update each reader to 2.1, but then (after readers are updated), all writers would need to go to 2.1 at once, to make sure they all "agree" that the write lock is now in the index.
        Hide
        Michael McCandless added a comment -

        Woops, I committed a silly violation of the LockFactory abstraction
        (I'm removing files in FSDirectory when that should only occur inside
        the LockFactory implementation.) I'm re-opening and I'll get the
        corrected fix in soon.

        Show
        Michael McCandless added a comment - Woops, I committed a silly violation of the LockFactory abstraction (I'm removing files in FSDirectory when that should only occur inside the LockFactory implementation.) I'm re-opening and I'll get the corrected fix in soon.
        Hide
        Michael McCandless added a comment -

        Closing all issues that were resolved for 2.1.

        Show
        Michael McCandless added a comment - Closing all issues that were resolved for 2.1.

          People

          • Assignee:
            Michael McCandless
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development