Derby
  1. Derby
  2. DERBY-2162

Shutting down a database loaded from a jar file via the classpath and URLClassLoader leaves an open file reference to the jar file containing the database.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 10.2.1.6, 10.3.1.4
    • Fix Version/s: 10.11.1.1
    • Component/s: Test
    • Urgency:
      Normal

      Description

      A bug in java.net.URLClassLoader causes the underlying jar file to be held open once a resource has been fetched and opened.
      Loading a class or just accessing the URL for the resource does not keep the jar open. Reported the bug to Sun, it is similar to the existing bug Java bug 4950148 but in this case no amount of garbage collection will close the jar.

      Derby exposes this as all containers/files are opened using as resources when loading the database from the classpath.

      On windows this is seen as the inability to delete the jar file, seen by the fixture testDatabaseInClasspath in DatabaseClassLoadingTest.

      Similar to DERBY-2083 but due to a different cause.

      1. d2162-1a.diff
        17 kB
        Knut Anders Hatlen

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open In Progress In Progress
          2665d 14h 52m 1 Knut Anders Hatlen 26/Mar/14 09:23
          In Progress In Progress Resolved Resolved
          23h 17m 1 Knut Anders Hatlen 27/Mar/14 08:40
          Resolved Resolved Closed Closed
          299d 15h 42m 1 Myrna van Lunteren 21/Jan/15 00:23
          Myrna van Lunteren made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Myrna van Lunteren added a comment -

          bulk change to close all issues resolved but not closed and not changed since June 1, 2014.

          Show
          Myrna van Lunteren added a comment - bulk change to close all issues resolved but not closed and not changed since June 1, 2014.
          Knut Anders Hatlen made changes -
          Component/s Test [ 11413 ]
          Component/s Services [ 11415 ]
          Hide
          Knut Anders Hatlen added a comment -

          Changing component from Services to Test, since the fix was to make the test close the URLClassLoader when it was done.

          Show
          Knut Anders Hatlen added a comment - Changing component from Services to Test, since the fix was to make the test close the URLClassLoader when it was done.
          Knut Anders Hatlen made changes -
          Status In Progress [ 3 ] Resolved [ 5 ]
          Issue & fix info Patch Available [ 10102 ]
          Fix Version/s 10.11.0.0 [ 12324243 ]
          Resolution Fixed [ 1 ]
          Hide
          ASF subversion and git services added a comment -

          Commit 1582220 from Knut Anders Hatlen in branch 'code/trunk'
          [ https://svn.apache.org/r1582220 ]

          DERBY-2162/DERBY-5618: Close the URLClassLoader when tests are done
          with it so that file handles are freed and the jar files can be
          deleted.

          Show
          ASF subversion and git services added a comment - Commit 1582220 from Knut Anders Hatlen in branch 'code/trunk' [ https://svn.apache.org/r1582220 ] DERBY-2162 / DERBY-5618 : Close the URLClassLoader when tests are done with it so that file handles are freed and the jar files can be deleted.
          Knut Anders Hatlen made changes -
          Issue & fix info Patch Available [ 10102 ]
          Knut Anders Hatlen made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Knut Anders Hatlen made changes -
          Attachment d2162-1a.diff [ 12636890 ]
          Hide
          Knut Anders Hatlen added a comment -

          The attached patch (d2162-1a.diff) makes ClasspathSetup.tearDown() call close() on the URLClassLoader to free the resources held by it. It rewrites DatabaseClassLoadingTest.testDatabaseInClasspath() so that it uses ClasspathSetup, and enables it on Java 7 and higher (not on Java 6 because it's missing URLClassLoader.close()).

          The patch also enables some previously disabled test cases in NativeAuthenticationServiceTest on Windows (except on Java 6).

          These test cases now run cleanly for me on Windows. Without the fix in ClasspathSetup.tearDown(), they fail because they cannot delete the jar files at the end of the test.

          URLClassLoader.close() is protected by a RuntimePermission. The patch grants that permission to derbyTesting.jar.

          Show
          Knut Anders Hatlen added a comment - The attached patch (d2162-1a.diff) makes ClasspathSetup.tearDown() call close() on the URLClassLoader to free the resources held by it. It rewrites DatabaseClassLoadingTest.testDatabaseInClasspath() so that it uses ClasspathSetup, and enables it on Java 7 and higher (not on Java 6 because it's missing URLClassLoader.close()). The patch also enables some previously disabled test cases in NativeAuthenticationServiceTest on Windows (except on Java 6). These test cases now run cleanly for me on Windows. Without the fix in ClasspathSetup.tearDown(), they fail because they cannot delete the jar files at the end of the test. URLClassLoader.close() is protected by a RuntimePermission. The patch grants that permission to derbyTesting.jar.
          Hide
          Knut Anders Hatlen added a comment -

          Another way to get the test case to pass, also on Java versions earlier than 7, is to call the URLConnection.setDefaultUseCaches() method first to disable caching of the jar files opened by URLClassLoader. That seems to make gc capable of freeing the file handle after the context class loader has been set to null at the end of the test case. An explicit call to System.gc() is needed before trying to delete the jar file, though, in order to make sure gc has run and released the file handle.

          This latter approach is probably too brittle to use in the regression tests, since System.gc() doesn't guarantee that all garbage is cleaned up. But I think these results support Dan's initial analysis that the file handle is kept open by the Java platform itself and not by Derby. I also think it means the solution for this problem must be implemented in the application layer (applications could either close the URLClassLoader when they are done with it, or disable caching of jar files). Having Derby close the URLClassLoader on shutdown is not an option, I think, as it is not possible for Derby to determine whether or not the application will still need the class loader after the Derby engine has been shut down.

          Show
          Knut Anders Hatlen added a comment - Another way to get the test case to pass, also on Java versions earlier than 7, is to call the URLConnection.setDefaultUseCaches() method first to disable caching of the jar files opened by URLClassLoader. That seems to make gc capable of freeing the file handle after the context class loader has been set to null at the end of the test case. An explicit call to System.gc() is needed before trying to delete the jar file, though, in order to make sure gc has run and released the file handle. This latter approach is probably too brittle to use in the regression tests, since System.gc() doesn't guarantee that all garbage is cleaned up. But I think these results support Dan's initial analysis that the file handle is kept open by the Java platform itself and not by Derby. I also think it means the solution for this problem must be implemented in the application layer (applications could either close the URLClassLoader when they are done with it, or disable caching of jar files). Having Derby close the URLClassLoader on shutdown is not an option, I think, as it is not possible for Derby to determine whether or not the application will still need the class loader after the Derby engine has been shut down.
          Knut Anders Hatlen made changes -
          Assignee Knut Anders Hatlen [ knutanders ]
          Hide
          Knut Anders Hatlen added a comment -

          Java 7 added a close() method to URLClassLoader for releasing the jar files. See https://bugs.openjdk.java.net/browse/JDK-4167874.

          If I enable testDatabaseInClasspath() and add a call to URLClassLoader.close() when it resets the context class loader, the test passes for me with JDK 7u51 on Windows. It fails to delete dclt.jar if I don't add the call to close().

          I'll try to get the test enabled permanently on Java 7 and higher using the new method.

          Show
          Knut Anders Hatlen added a comment - Java 7 added a close() method to URLClassLoader for releasing the jar files. See https://bugs.openjdk.java.net/browse/JDK-4167874 . If I enable testDatabaseInClasspath() and add a call to URLClassLoader.close() when it resets the context class loader, the test passes for me with JDK 7u51 on Windows. It fails to delete dclt.jar if I don't add the call to close(). I'll try to get the test enabled permanently on Java 7 and higher using the new method.
          Knut Anders Hatlen made changes -
          Link This issue is duplicated by DERBY-5618 [ DERBY-5618 ]
          Gavin made changes -
          Workflow jira [ 12391823 ] Default workflow, editable Closed status [ 12798777 ]
          Kathey Marsden made changes -
          Labels derby_triage10_5_2
          Knut Anders Hatlen made changes -
          Urgency Normal
          Hide
          Knut Anders Hatlen added a comment -

          Triaged for 10.5.2.

          Show
          Knut Anders Hatlen added a comment - Triaged for 10.5.2.
          Kathey Marsden made changes -
          Assignee Daniel John Debrunner [ djd ]
          Mike Matrigali made changes -
          Field Original Value New Value
          Component/s Services [ 11415 ]
          Daniel John Debrunner created issue -

            People

            • Assignee:
              Knut Anders Hatlen
              Reporter:
              Daniel John Debrunner
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development