Lucene - Core
  1. Lucene - Core
  2. LUCENE-3139

LuceneTestCase.afterClass does not print enough information if a temp-test-dir fails to delete

    Details

    • Type: Test Test
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2, 4.0-ALPHA
    • Component/s: general/test
    • Labels:
      None
    • Lucene Fields:
      New, Patch Available

      Description

      I've hit an exception from LTC.afterClass when _TestUtil.rmDir failed (on write.lock, as if some test did not release resources). However, I had no idea which test caused that (i.e. opened the temp directory and did not release resources).

      I think we should do the following:

      • Track in LTC a map from dirName -> StackTraceElement
      • In afterClass if _TestUtil.rmDir fails, print the STE of that particular dir, so we know where was this directory created from
      • Make tempDirs private and create accessor method, so that we control the inserts to this map (today the Set is updated by LTC, _TestUtils and TestBackwards !)
      1. LUCENE-3139.patch
        15 kB
        Shai Erera
      2. LUCENE-3139.patch
        8 kB
        Shai Erera
      3. LUCENE-3139.patch
        4 kB
        Shai Erera

        Activity

        Hide
        Robert Muir added a comment -

        +1

        Show
        Robert Muir added a comment - +1
        Hide
        Shai Erera added a comment -

        Perhaps we should also fail the test if that happens? Was there reason why only the stacktrace printed, but tests were considered successful?

        Show
        Shai Erera added a comment - Perhaps we should also fail the test if that happens? Was there reason why only the stacktrace printed, but tests were considered successful?
        Hide
        Robert Muir added a comment -

        some tests are still problematic, at least on windows... I think perhaps some of the crazier ones like DiskFull, TestCrash, anything that has to disable MockDirectoryWrappers's checks because they must create corrupt indexes or other scary things.

        Show
        Robert Muir added a comment - some tests are still problematic, at least on windows... I think perhaps some of the crazier ones like DiskFull, TestCrash, anything that has to disable MockDirectoryWrappers's checks because they must create corrupt indexes or other scary things.
        Hide
        Shai Erera added a comment -

        Patch adds registerTempFile to LTC plus prints stack information if rmDir fails.

        I think we should also fail the test if that happens?

        Show
        Shai Erera added a comment - Patch adds registerTempFile to LTC plus prints stack information if rmDir fails. I think we should also fail the test if that happens?
        Hide
        Shai Erera added a comment -

        some tests are still problematic, at least on windows...

        Ok, I didn't notice your comment when posted the patch. So let's keep it as-is.

        I think it's ready to commit

        Show
        Shai Erera added a comment - some tests are still problematic, at least on windows... Ok, I didn't notice your comment when posted the patch. So let's keep it as-is. I think it's ready to commit
        Hide
        Shai Erera added a comment -

        Patch applies the same changes to backwards' LTC.

        Show
        Shai Erera added a comment - Patch applies the same changes to backwards' LTC.
        Hide
        Shai Erera added a comment -

        When I run 'ant test-backwards' I see these exceptions:

           [junit] ------------- Standard Error -----------------
           [junit] java.io.IOException: could not delete D:\dev\lucene\lucene-3x\lucene\build\backwards\test\1\test4293913517498927234tmp\_1.fdt
           [junit]     at org.apache.lucene.util._TestUtil.rmDir(_TestUtil.java:65)
           [junit]     at org.apache.lucene.util.LuceneTestCase.afterClassLuceneTestCaseJ4(LuceneTestCase.java:291)
           [junit]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
           [junit]     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
           [junit]     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
           [junit]     at java.lang.reflect.Method.invoke(Method.java:611)
           [junit]     at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
           [junit]     at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
           [junit]     at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
           [junit]     at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
           [junit]     at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
           [junit]     at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
           [junit]     at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
           [junit]     at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
           [junit]     at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:743)
           [junit] path D:\dev\lucene\lucene-3x\lucene\build\backwards\test\1\test4293913517498927234tmp allocated from
           [junit] org.apache.lucene.util.LuceneTestCase.registerTempFile(LuceneTestCase.java:930)
           [junit] org.apache.lucene.util.LuceneTestCase.newDirectoryImpl(LuceneTestCase.java:945)
           [junit] org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:733)
           [junit] org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:725)
           [junit] org.apache.lucene.index.TestIndexWriterWithThreads._testMultipleThreadsFailure(TestIndexWriterWithThreads.java:212)
           [junit] org.apache.lucene.index.TestIndexWriterWithThreads.testIOExceptionDuringWriteSegmentWithThreads(TestIndexWriterWithThreads.java:381)
        

        and

            [junit] ------------- Standard Error -----------------
            [junit] java.io.IOException: could not delete D:\dev\lucene\lucene-3x\lucene\build\backwards\test\5\test6976265647485126574tmp\write.lock
            [junit]     at org.apache.lucene.util._TestUtil.rmDir(_TestUtil.java:65)
            [junit]     at org.apache.lucene.util.LuceneTestCase.afterClassLuceneTestCaseJ4(LuceneTestCase.java:291)
            [junit]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            [junit]     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
            [junit]     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
            [junit]     at java.lang.reflect.Method.invoke(Method.java:611)
            [junit]     at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
            [junit]     at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
            [junit]     at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
            [junit]     at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
            [junit]     at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
            [junit]     at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
            [junit]     at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
            [junit]     at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
            [junit]     at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:743)
            [junit] path D:\dev\lucene\lucene-3x\lucene\build\backwards\test\5\test6976265647485126574tmp allocated from
            [junit] org.apache.lucene.util.LuceneTestCase.registerTempFile(LuceneTestCase.java:930)
            [junit] org.apache.lucene.util.LuceneTestCase.newDirectoryImpl(LuceneTestCase.java:945)
            [junit] org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:733)
            [junit] org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:725)
            [junit] org.apache.lucene.index.TestIndexReader.testReopenChangeReadonly(TestIndexReader.java:1622)
        

        and

            [junit] ------------- Standard Error -----------------
            [junit] java.io.IOException: could not delete D:\dev\lucene\lucene-3x\lucene\build\backwards\test\5\test7434165808901493103tmp\write.lock
            [junit]     at org.apache.lucene.util._TestUtil.rmDir(_TestUtil.java:65)
            [junit]     at org.apache.lucene.util.LuceneTestCase.afterClassLuceneTestCaseJ4(LuceneTestCase.java:291)
            [junit]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            [junit]     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
            [junit]     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
            [junit]     at java.lang.reflect.Method.invoke(Method.java:611)
            [junit]     at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
            [junit]     at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
            [junit]     at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
            [junit]     at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
            [junit]     at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
            [junit]     at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
            [junit]     at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
            [junit]     at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
            [junit]     at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:743)
            [junit] path D:\dev\lucene\lucene-3x\lucene\build\backwards\test\5\test7434165808901493103tmp allocated from
            [junit] org.apache.lucene.util.LuceneTestCase.registerTempFile(LuceneTestCase.java:930)
            [junit] org.apache.lucene.util.LuceneTestCase.newDirectoryImpl(LuceneTestCase.java:945)
            [junit] org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:733)
            [junit] org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:725)
            [junit] org.apache.lucene.index.TestPersistentSnapshotDeletionPolicy.getDeletionPolicy(TestPersistentSnapshotDeletionPolicy.java:53)
            [junit] org.apache.lucene.index.TestSnapshotDeletionPolicy.testReleaseSnapshot(TestSnapshotDeletionPolicy.java:336)
        

        and

            [junit] ------------- Standard Error -----------------
            [junit] java.io.IOException: could not delete D:\dev\lucene\lucene-3x\lucene\build\backwards\test\7\test4762963085363551376tmp\_a.fnm
            [junit]     at org.apache.lucene.util._TestUtil.rmDir(_TestUtil.java:65)
            [junit]     at org.apache.lucene.util.LuceneTestCase.afterClassLuceneTestCaseJ4(LuceneTestCase.java:291)
            [junit]     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            [junit]     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
            [junit]     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
            [junit]     at java.lang.reflect.Method.invoke(Method.java:611)
            [junit]     at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
            [junit]     at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
            [junit]     at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
            [junit]     at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37)
            [junit]     at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
            [junit]     at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
            [junit]     at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
            [junit]     at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
            [junit]     at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:743)
            [junit] path D:\dev\lucene\lucene-3x\lucene\build\backwards\test\7\test4762963085363551376tmp allocated from
            [junit] org.apache.lucene.util.LuceneTestCase.registerTempFile(LuceneTestCase.java:930)
            [junit] org.apache.lucene.util.LuceneTestCase.newDirectoryImpl(LuceneTestCase.java:945)
            [junit] org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:733)
            [junit] org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:725)
            [junit] org.apache.lucene.TestMergeSchedulerExternal.testSubclassConcurrentMergeScheduler(TestMergeSchedulerExternal.java:85)
        

        It's weird - even simple classes fail ... maybe the random dir name we make up for the test is not random enough on my machine? (It is known for its ability to pick really interesting seeds for Lucene tests ).

        Perhaps we should have the test method in the dirname, in addition to Random?

        I see _TestUtil.getTempDir does this:

          public static File getTempDir(String desc) {
            File f = new File(LuceneTestCase.TEMP_DIR, desc + "." + new Random().nextLong());
            LuceneTestCase.registerTempFile(f);
            return f;
          }
        

        That's bad no? Shouldn't it use LTC.random? And then we could add that seed to the print in LTC.afterClass ...?

        Show
        Shai Erera added a comment - When I run 'ant test-backwards' I see these exceptions: [junit] ------------- Standard Error ----------------- [junit] java.io.IOException: could not delete D:\dev\lucene\lucene-3x\lucene\build\backwards\test\1\test4293913517498927234tmp\_1.fdt [junit] at org.apache.lucene.util._TestUtil.rmDir(_TestUtil.java:65) [junit] at org.apache.lucene.util.LuceneTestCase.afterClassLuceneTestCaseJ4(LuceneTestCase.java:291) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) [junit] at java.lang.reflect.Method.invoke(Method.java:611) [junit] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) [junit] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) [junit] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) [junit] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37) [junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:236) [junit] at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:743) [junit] path D:\dev\lucene\lucene-3x\lucene\build\backwards\test\1\test4293913517498927234tmp allocated from [junit] org.apache.lucene.util.LuceneTestCase.registerTempFile(LuceneTestCase.java:930) [junit] org.apache.lucene.util.LuceneTestCase.newDirectoryImpl(LuceneTestCase.java:945) [junit] org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:733) [junit] org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:725) [junit] org.apache.lucene.index.TestIndexWriterWithThreads._testMultipleThreadsFailure(TestIndexWriterWithThreads.java:212) [junit] org.apache.lucene.index.TestIndexWriterWithThreads.testIOExceptionDuringWriteSegmentWithThreads(TestIndexWriterWithThreads.java:381) and [junit] ------------- Standard Error ----------------- [junit] java.io.IOException: could not delete D:\dev\lucene\lucene-3x\lucene\build\backwards\test\5\test6976265647485126574tmp\write.lock [junit] at org.apache.lucene.util._TestUtil.rmDir(_TestUtil.java:65) [junit] at org.apache.lucene.util.LuceneTestCase.afterClassLuceneTestCaseJ4(LuceneTestCase.java:291) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) [junit] at java.lang.reflect.Method.invoke(Method.java:611) [junit] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) [junit] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) [junit] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) [junit] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37) [junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:236) [junit] at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:743) [junit] path D:\dev\lucene\lucene-3x\lucene\build\backwards\test\5\test6976265647485126574tmp allocated from [junit] org.apache.lucene.util.LuceneTestCase.registerTempFile(LuceneTestCase.java:930) [junit] org.apache.lucene.util.LuceneTestCase.newDirectoryImpl(LuceneTestCase.java:945) [junit] org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:733) [junit] org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:725) [junit] org.apache.lucene.index.TestIndexReader.testReopenChangeReadonly(TestIndexReader.java:1622) and [junit] ------------- Standard Error ----------------- [junit] java.io.IOException: could not delete D:\dev\lucene\lucene-3x\lucene\build\backwards\test\5\test7434165808901493103tmp\write.lock [junit] at org.apache.lucene.util._TestUtil.rmDir(_TestUtil.java:65) [junit] at org.apache.lucene.util.LuceneTestCase.afterClassLuceneTestCaseJ4(LuceneTestCase.java:291) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) [junit] at java.lang.reflect.Method.invoke(Method.java:611) [junit] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) [junit] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) [junit] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) [junit] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37) [junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:236) [junit] at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:743) [junit] path D:\dev\lucene\lucene-3x\lucene\build\backwards\test\5\test7434165808901493103tmp allocated from [junit] org.apache.lucene.util.LuceneTestCase.registerTempFile(LuceneTestCase.java:930) [junit] org.apache.lucene.util.LuceneTestCase.newDirectoryImpl(LuceneTestCase.java:945) [junit] org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:733) [junit] org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:725) [junit] org.apache.lucene.index.TestPersistentSnapshotDeletionPolicy.getDeletionPolicy(TestPersistentSnapshotDeletionPolicy.java:53) [junit] org.apache.lucene.index.TestSnapshotDeletionPolicy.testReleaseSnapshot(TestSnapshotDeletionPolicy.java:336) and [junit] ------------- Standard Error ----------------- [junit] java.io.IOException: could not delete D:\dev\lucene\lucene-3x\lucene\build\backwards\test\7\test4762963085363551376tmp\_a.fnm [junit] at org.apache.lucene.util._TestUtil.rmDir(_TestUtil.java:65) [junit] at org.apache.lucene.util.LuceneTestCase.afterClassLuceneTestCaseJ4(LuceneTestCase.java:291) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [junit] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60) [junit] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) [junit] at java.lang.reflect.Method.invoke(Method.java:611) [junit] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) [junit] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) [junit] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) [junit] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:37) [junit] at org.junit.runners.ParentRunner.run(ParentRunner.java:236) [junit] at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) [junit] at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:743) [junit] path D:\dev\lucene\lucene-3x\lucene\build\backwards\test\7\test4762963085363551376tmp allocated from [junit] org.apache.lucene.util.LuceneTestCase.registerTempFile(LuceneTestCase.java:930) [junit] org.apache.lucene.util.LuceneTestCase.newDirectoryImpl(LuceneTestCase.java:945) [junit] org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:733) [junit] org.apache.lucene.util.LuceneTestCase.newDirectory(LuceneTestCase.java:725) [junit] org.apache.lucene.TestMergeSchedulerExternal.testSubclassConcurrentMergeScheduler(TestMergeSchedulerExternal.java:85) It's weird - even simple classes fail ... maybe the random dir name we make up for the test is not random enough on my machine? (It is known for its ability to pick really interesting seeds for Lucene tests ). Perhaps we should have the test method in the dirname, in addition to Random? I see _TestUtil.getTempDir does this: public static File getTempDir( String desc) { File f = new File(LuceneTestCase.TEMP_DIR, desc + "." + new Random().nextLong()); LuceneTestCase.registerTempFile(f); return f; } That's bad no? Shouldn't it use LTC.random? And then we could add that seed to the print in LTC.afterClass ...?
        Hide
        Shai Erera added a comment -

        I think I've found the problem - MockIndexOutputWrapper did not close delegate if dir.maybeThrowEx actually threw an exception. Here's a patch that fixes it:

        Index: lucene/src/test-framework/org/apache/lucene/store/MockIndexOutputWrapper.java
        ===================================================================
        --- lucene/src/test-framework/org/apache/lucene/store/MockIndexOutputWrapper.java       (revision 1127062)
        +++ lucene/src/test-framework/org/apache/lucene/store/MockIndexOutputWrapper.java       (working copy)
        @@ -45,20 +45,23 @@
        
           @Override
           public void close() throws IOException {
        -    dir.maybeThrowDeterministicException();
        -    delegate.close();
        -    if (dir.trackDiskUsage) {
        -      // Now compute actual disk usage & track the maxUsedSize
        -      // in the MockDirectoryWrapper:
        -      long size = dir.getRecomputedActualSizeInBytes();
        -      if (size > dir.maxUsedSize) {
        -        dir.maxUsedSize = size;
        +    try {
        +      dir.maybeThrowDeterministicException();
        +    } finally {
        +      delegate.close();
        +      if (dir.trackDiskUsage) {
        +        // Now compute actual disk usage & track the maxUsedSize
        +        // in the MockDirectoryWrapper:
        +        long size = dir.getRecomputedActualSizeInBytes();
        +        if (size > dir.maxUsedSize) {
        +          dir.maxUsedSize = size;
        +        }
               }
        +      synchronized(dir) {
        +        dir.openFileHandles.remove(this);
        +        dir.openFilesForWrite.remove(name);
        +      }
             }
        -    synchronized(dir) {
        -      dir.openFileHandles.remove(this);
        -      dir.openFilesForWrite.remove(name);
        -    }
           }
        
           @Override
        

        Maybe we solve it by moving delegate.close() before dir.maybeThrow, instead of the try-finally?

        Show
        Shai Erera added a comment - I think I've found the problem - MockIndexOutputWrapper did not close delegate if dir.maybeThrowEx actually threw an exception. Here's a patch that fixes it: Index: lucene/src/test-framework/org/apache/lucene/store/MockIndexOutputWrapper.java =================================================================== --- lucene/src/test-framework/org/apache/lucene/store/MockIndexOutputWrapper.java (revision 1127062) +++ lucene/src/test-framework/org/apache/lucene/store/MockIndexOutputWrapper.java (working copy) @@ -45,20 +45,23 @@ @Override public void close() throws IOException { - dir.maybeThrowDeterministicException(); - delegate.close(); - if (dir.trackDiskUsage) { - // Now compute actual disk usage & track the maxUsedSize - // in the MockDirectoryWrapper: - long size = dir.getRecomputedActualSizeInBytes(); - if (size > dir.maxUsedSize) { - dir.maxUsedSize = size; + try { + dir.maybeThrowDeterministicException(); + } finally { + delegate.close(); + if (dir.trackDiskUsage) { + // Now compute actual disk usage & track the maxUsedSize + // in the MockDirectoryWrapper: + long size = dir.getRecomputedActualSizeInBytes(); + if (size > dir.maxUsedSize) { + dir.maxUsedSize = size; + } } + synchronized (dir) { + dir.openFileHandles.remove( this ); + dir.openFilesForWrite.remove(name); + } } - synchronized (dir) { - dir.openFileHandles.remove( this ); - dir.openFilesForWrite.remove(name); - } } @Override Maybe we solve it by moving delegate.close() before dir.maybeThrow, instead of the try-finally?
        Hide
        Shai Erera added a comment -

        Patch fixes MockIndexOutputWrapper to always call delegate.close().

        On the way, I fixed TestPersistentSnapshotDeletionPolicy to always close the PSDP, as well as fixed a "leak" in PSDP itself.

        I will commit this one. There are still tests that fail to delete directories, but I think we should iterate on them separately, as all seem to be related to how MockIOWrapper handles those maybeThrowDeterminsticEx. I'll send a separate mail.

        This issue is about fixing LTC so that it outputs more information about directories that fail to delete. You can simulate failures, on Windows, by running tests w/ -Dtests.directory=FSDirectory, then track the stacktraces that are printed.

        Show
        Shai Erera added a comment - Patch fixes MockIndexOutputWrapper to always call delegate.close(). On the way, I fixed TestPersistentSnapshotDeletionPolicy to always close the PSDP, as well as fixed a "leak" in PSDP itself. I will commit this one. There are still tests that fail to delete directories, but I think we should iterate on them separately, as all seem to be related to how MockIOWrapper handles those maybeThrowDeterminsticEx. I'll send a separate mail. This issue is about fixing LTC so that it outputs more information about directories that fail to delete. You can simulate failures, on Windows, by running tests w/ -Dtests.directory=FSDirectory, then track the stacktraces that are printed.
        Hide
        Shai Erera added a comment -

        Committed revision 1127470 (trunk).
        Committed revision 1127471 (3x).

        LTC now verbose whatever we need to pursue rmDir failures.

        Show
        Shai Erera added a comment - Committed revision 1127470 (trunk). Committed revision 1127471 (3x). LTC now verbose whatever we need to pursue rmDir failures.
        Hide
        Robert Muir added a comment -

        Bulk closing for 3.2

        Show
        Robert Muir added a comment - Bulk closing for 3.2

          People

          • Assignee:
            Shai Erera
            Reporter:
            Shai Erera
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development