Lucene - Core
  1. Lucene - Core
  2. LUCENE-1669

MMapDirectory on Windows silently fails to write to a file if also open for read

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Invalid
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: core/store
    • Labels:
      None
    • Lucene Fields:
      New

      Description

      This is not a normal situation Lucene currently encounters, though we
      are discussing exactly this possibility in LUCENE-1313.

      I only hit it in digging down on a test failure in LUCENE-1658:

          [junit] Testcase: testIndexAndMerge(org.apache.lucene.index.TestDoc):	FAILED
          [junit] junit.framework.AssertionFailedError:
          [junit] 	at org.apache.lucene.index.FieldsWriter.addRawDocuments(FieldsWriter.java:249)
          [junit] 	at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:350)
          [junit] 	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139)
          [junit] 	at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:116)
          [junit] 	at org.apache.lucene.index.TestDoc.merge(TestDoc.java:182)
          [junit] 	at org.apache.lucene.index.TestDoc.testIndexAndMerge(TestDoc.java:117)
          [junit] 	at org.apache.lucene.util.LuceneTestCase.runTest(LuceneTestCase.java:88)
      

      That failure happens on Windows 64bit, if you use MMapDirectory, and
      the index is on a remote (CIFS) mount.

      This test opens a SegmentReader against doc store files that
      IndexWriter still has open.

      I whittled it down to this test showing the root cause:

      public void testMMapWriteRead() throws Exception {
          MMapDirectory dir = new MMapDirectory(new File("readwrite"), null);
          //NIOFSDirectory dir = new NIOFSDirectory(new File("readwrite"), null);
          //SimpleFSDirectory dir = new SimpleFSDirectory(new File("readwrite"), null);
          IndexOutput out = dir.createOutput("one");
          out.writeLong(17);
          out.flush();
          // open the same file we are writing
          dir.openInput("one").close();
          // write another long
          out.writeLong(56);
          out.close();
          IndexInput in = dir.openInput("one");
          assertEquals(17, in.readLong());
          assertEquals(56, in.readLong());
          in.close();
      }
      

      NIOFSDir and SimpleFSDir pass the test fine, which is nice to know
      (for LUCENE-1313). MMapDir passes fine on a local drive, but fails on
      a remote CIFS mount.

      I'm not sure what's going on. It seems like it could be a JRE bug –
      behavior shouldn't change on local vs remote drive. I've only tested
      it with java 1.6.0_11.

        Activity

        Hide
        Uwe Schindler added a comment -

        In the mmap bug report (we know it) at http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4724038 one user states:

        Submitted On 13-MAR-2006
        sebastien.vauclair
        I am using the workaround shown bellow to unmap buffers.

        Everything works fine with multiple applications reading from the same growing local file, but there is a problem if the file is accessed using a Windows UNC path like "//machine/c$/dir/file": Buffer.get() returns 0 for some bytes near EOF in such files.

        This bug is reproductible even with 127.0.0.1 as the machine.

        Should I open a new bug? I was not sure considering the workaround used to unmap has not been officially approved.

        Note: adding a 1 second sleep after the buffer has been GC'd fixes the problem - but that's of course not always acceptable.

        Maybe this is the cause?

        Show
        Uwe Schindler added a comment - In the mmap bug report (we know it) at http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4724038 one user states: Submitted On 13-MAR-2006 sebastien.vauclair I am using the workaround shown bellow to unmap buffers. Everything works fine with multiple applications reading from the same growing local file, but there is a problem if the file is accessed using a Windows UNC path like "//machine/c$/dir/file": Buffer.get() returns 0 for some bytes near EOF in such files. This bug is reproductible even with 127.0.0.1 as the machine. Should I open a new bug? I was not sure considering the workaround used to unmap has not been officially approved. Note: adding a 1 second sleep after the buffer has been GC'd fixes the problem - but that's of course not always acceptable. Maybe this is the cause?
        Hide
        Michael McCandless added a comment -

        That sounds exactly like what I was hitting – getting 0 from readLong instead of the expected 56, in the above test. Except, my test fails even before doing the hack... so it's not the hack that's causing it.

        Show
        Michael McCandless added a comment - That sounds exactly like what I was hitting – getting 0 from readLong instead of the expected 56, in the above test. Except, my test fails even before doing the hack... so it's not the hack that's causing it.
        Hide
        Erick Erickson added a comment -

        SPRING_CLEANING_2013. We can reopen if there's interest. Guessing this has been handled for a long time.

        Show
        Erick Erickson added a comment - SPRING_CLEANING_2013. We can reopen if there's interest. Guessing this has been handled for a long time.

          People

          • Assignee:
            Unassigned
            Reporter:
            Michael McCandless
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development