[PATCH] Lock Framework - allows custom lock mechanism

• Key: LUCENE-305
• Type: Improvement
• Status: Closed
• Resolution: Duplicate
• Priority: Minor
• Assignee: Yonik Seeley
• Reporter: Jeff Patterson
• Component/s: Store
• Affects Version/s: unspecified
• Fix Version/s: 2.1

Environment

Operating System: other
Platform: All

Description

Proposal: Pluggable Lock Framework for Lucene
Date: Nov 2004
Developer: Jeff Patterson (jeffATwebdoyen.com - http://www.webdoyen.com)

------

Abstract: A framework to allow Lucene users to override the default
FileSystem locking mechanism with a custom lock mechanism.

A Lucene user may develop a new class that extends
org.apache.lucene.store.Lock and implement bodies for the following
methods:
public boolean obtain() - to obtain custom lock
public boolean isLocked() - to detect custom lock
public void release() - to release custom lock

NOTE: When implementing these methods, the developer should make sure to
use the this.getLockName() method on the Lock to identify which lock
is being manipulated (see Modified Files below for more).

After developed, the new class must be added to the classpath (along
with any other supporting classes/libraries needed by the new class),
and the Lucene framework must be alerted of the new class by way of
the "org.apache.lucene.lockClass" -D System property. Example:

java -Dorg.apache.lucene.lockClass=foo.MyCustomLocker LuceneTest

------

Modified Files: The following files were modified to support
this framework (DIFF files at end):

• org.apache.lucene.store.Lock
  The member "lockName" and an accompanying protected getter and
  setter were added to this class to support naming the lock. This
  is transparent to the default lock mechanism and is only useful
  when writing a custom lock.

• org.apache.lucene.store.FSDirectory
  Instead of instantiating a default Lock, this class now checks
  to see if an overridden Lock mechanism is provided, and if so
  asks the LockFactory (see below) to provide an overridden Lock
  class.

New Files: The following files were added to support this framework:

• org.apache.lucene.store.LockFactory
  This class is used to reflect and instantiate by name the custom
  Lock implementation. Error handing should be modified in this
  class, but that would have required a more extensive code overhaul.
  The javadocs for the LockFactory contain a skeleton Java file for
  a custom lock implementation.

------

Comments

[Jeff Patterson]

Attached find the modified files as well as the new file
LockFactory and a demo lock implementation. I also am
attaching the DIFF files for the 2 modified files...

[Jeff Patterson]

Created an attachment (id=13369)
DIFF file for FSDirectory.java

Created an attachment (id=13370)
Modified FSDirectory.java file

[Jeff Patterson]

Created an attachment (id=13371)
DIFF file for Lock.java

[Jeff Patterson]

Created an attachment (id=13372)
Modified Lock.java file

[Jeff Patterson]

Created an attachment (id=13373)
New LockFactory.java class

[Jeff Patterson]

Created an attachment (id=13374)
Demo custom lock mechanism (not necessary for framework, but demonstrates framwork usage)

[Michael McCandless]

I'm working towards a patch for this. I changed the name of the issue
to better reflect the goal.

I started with the above approach and made some changes / brought it
up to current Lucene HEAD sources.

I'd like to sanity check the approach to get any feedback:

• Create a new abstract base class called LockFactory (like the
  above patch). Purpose of this class is to provide a makeLock()
  method that returns a Lock object for given name.
  Directory.makeLock just calls this method.

• Create 3 concrete subclasses (for now):

• SimpleFSLockFactory (factored out of FSDirectory)

• SingleProcessLockFactory (factored out of RAMDirectory)

• NoLockFactory

• (then, some time soon: NativeFSLockFactory)

• In Directory, add a setLockFactory() method. This method is
  optional. If it is not called, then each Directory has a default
  LockFactory that's used instead. FSDirectory defaults to either
  SimpleFSLockFactory or NoLockFactory (if setDisabledLocks are
  called) and RAMDirectory defaults to SingleProcessLockFactory.

I believe this approach will be fully backwards compatible with
current locking/APIs:

• FSDirectory and RAMDirectory (and anyone who subclasses) use the
  same locking they use today. Same lock file names are used and
  same defaulting of LOCK_DIR is used.

• If any subclass of Directory has its own makeLock() method, it
  will still fully override all of this new logic.

• No existing APIs are changed; only new methods are added and new
  classes (*LockFactory).

• Performance should be the same (I'm trying hard not to cause
  additional code to be executed in the makeLock() code path). I'll
  test this to make sure I'm right

This approach means that a user can instantiate a Directory, any
Directory, and then set its locking implementation like this:

Directory.setLockFactory(new SingleProcessLockFactory();

So this should make it easy to use an FSDirectory with a SingleProcess
lock, for example. Users can also subclass their own LockFactory
implementations (eg the MySQL based locking above) and easily use
those for locking.

Please note that the goal here is to make Locking independent of the
Directory, meaning I'm not yet adding any new locking implementations.
The above locking implementations were already pre-existing in Lucene,
just refactored to subclasses of LockFactory. After this (I'll open a
separate issue), I'd like to build a NativeFSLockFactory to address
the original issues/problems we've seen with the
java.io.File.createNewFile() implementation we have now.

This approach differs from the original approach above in that instead
of a global property that sets lock factory class, you explicity call
setLockFactory for a given Directory instance. In addition, this
approach factors out all locking logic (except
getDefaultLockFactory()) inside FSDirectory and RAMDirectory into
LockFactory.

Please let me know if anyone has any concerns / issues with this plan!
I have an initial working version that passes all current unit tests,
but I still need to add some more specific test cases for locking, do
performance testing, etc. Thanks.

[Michael McCandless]

See LUCENE-635 which subsumes this patch.

[Michael McCandless]

I think we can close this issue now that LUCENE-635 is resolved.