[LUCENENET-618] Need a Cross-OS NativeFSLock Implementation

• Type: Task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: Lucene.Net 4.8.0
• Fix Version/s: Lucene.Net 4.8.0
• Component/s: Lucene.Net Core
• Labels: cross-platform linux macOS
• Flags: Important
• Assignee: Shad Storhaug
• Reporter: Shad Storhaug
• Created: 15/Aug/19 07:59
• Updated: 26/Oct/19 19:08
• Resolved: 26/Oct/19 19:08

Description

Now that we have (finally) begun testing on platforms other than Windows, it has been discovered that the NativeFSLock doesn't work reliably on other platforms.

Due to differences in how Java and .NET handle file system locking, we have switched from the Lucene 4.8.0 implementation to one that is based on FileStream.Lock. While this implementation does include support for .NET Framework/.NET Standard 1.x/.NET Standard 2.x, it relies on HResult error codes that exist only on Windows.

I have done some research, and it turns out that using HResult error codes cannot be made reliably portable across platforms. So, I used the original implementation we had as a fallback when the OS is not Windows.

        internal virtual Lock NewLock(string path)
        {
            if (Constants.WINDOWS)
                return new WindowsNativeFSLock(this, m_lockDir, path);

            // Fallback implementation for unknown platforms that don't rely on HResult
            return new NativeFSLock(this, m_lockDir, path);
        }

Unfortunately, we are now seeing error messages when testing on macOS and Linux, which are a clear symptom of resource locking failures:

System.IO.IOException : The process cannot access the file '/tmp/LuceneTemp/index-MMapDirectory-i0xc4fz2/write.lock' because it is being used by another process.

at Lucene.Net.Util.IOUtils.DisposeWhileHandlingException(Exception priorException, IDisposable[] objects) in D:\a\1\s\src\Lucene.Net\Util\IOUtils.cs:line 197at Lucene.Net.Store.Directory.Copy(Directory to, String src, String dest, IOContext context) in D:\a\1\s\src\Lucene.Net\Store\Directory.cs:line 214at Lucene.Net.Store.MockDirectoryWrapper.Copy(Directory to, String src, String dest, IOContext context) in D:\a\1\s\src\Lucene.Net.TestFramework\Store\MockDirectoryWrapper.cs:line 1320at Lucene.Net.Store.RAMDirectory..ctor(Directory dir, Boolean closeDir, IOContext context) in D:\a\1\s\src\Lucene.Net\Store\RAMDirectory.cs:line 106at Lucene.Net.Index.TestTermVectorsWriter.TestTermVectorCorruption() in D:\a\1\s\src\Lucene.Net.Tests\Index\TestTermVectorsWriter.cs:line 446

It turns out the implementation that was used as a fallback had been contributed back when .NET Framework was the only target framework in Lucene.Net and it sounds like running on non-Windows platforms was not being considered in its implementation.

As a result, we only have 2 implementations that work on Windows and none that work reliably on other platforms.

The NativeFSLock implementation does prevent the "file in use" error from happening frequently on non-Windows platforms, but doesn't prevent it from happening completely. However, the WindowsNativeFSLock is reliably making all of the tests pass on Windows across several dozen test runs.

IMO, the approach being used on Windows is fine, but for other operating systems we should fallback to an implementation that is more reliable than the one we currently have.