Wicket
  1. Wicket
  2. WICKET-438

File handles are leaked when loading images from a jar file, Tomcat crashes

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.5
    • Fix Version/s: 1.2.6, 1.3.0-beta1
    • Component/s: wicket
    • Labels:
      None
    • Environment:
      java version "1.5.0_07", Mac OS X 10.4, Tomcat 5.5.20
      and also
      java version "1.5.0_07", Linux 2.6.8, Tomcat 5.5.17
      Firefox 2.0.3 with Web developer Toolbar

      Description

      Wicket leaks file handles when it accesses image files that are part of a jar file, e.g. wicket-1.2.5.jar.

      To reproduce this bug, please switch off the browser cache.

      My Wicket application uses TreeTable. When I access the page with the TreeTable on my Mac server, I see the number of open file handles jump. If I do that repeatedly, I can leak about 50 file handles.

      I observe the number of open file handles via this shell command: while true; do sleep 1; lsof | grep wicket | wc -l; done

      If I do the same test on a Linux server, the number of open files jumps so quickly that I reach 1015 open files after about 4 requests. At this point, Tomcat runs out of file handles, writes a log entry in catalina.out and dies. Probably Tomcat shouldn't crash, but the problem is that Wicket leaks file handles.

      I can also see leaked file handles when I repeatedly access an image from a jar file via http://localhost:8070/wicket-examples/ajax/resources/wicket.extensions.markup.html.tree.DefaultAbstractTree/res/tree-images.png

      The problem can also be reproduced via wicket-examples 1.2.5 and its TreeTable example page by exanding and collapsing the tree table repeatedly.

      If I unpack both wicket-1.2.5.jar and wicket-extensions-1.2.5.jar in WEB-INF/classes and delete the jars afterwards, the number of used file handles remains constant at 5! So access to images in WEB-INF/classes works fine.

      "lsof | grep wicket" gives me:

      firefox-b 1372 stefan 19r REG 14,2 486664 1733273 /Users/stefan/Software/apache-tomcat-5.5.20/webapps/middletier/WEB-INF/lib/wicket-extensions-1.2.5.jar
      bash 1880 stefan cwd DIR 14,2 578 1058285 /Users/stefan/Software/wicket-1.2.5
      bash 4339 stefan cwd DIR 14,2 578 1058285 /Users/stefan/Software/wicket-1.2.5
      java 9529 stefan txt REG 14,2 1249210 1750285 /Users/stefan/Software/apache-tomcat-5.5.20/webapps/middletier/WEB-INF/lib/wicket-1.2.zoomos-SNAPSHOT.jar
      java 9529 stefan txt REG 14,2 486664 1750274 /Users/stefan/Software/apache-tomcat-5.5.20/webapps/middletier/WEB-INF/lib/wicket-extensions-1.2.5.jar
      java 9529 stefan txt REG 14,2 11063 1750305 /Users/stefan/Software/apache-tomcat-5.5.20/webapps/middletier/WEB-INF/lib/wicket-spring-annot-1.2.5.jar
      java 9529 stefan txt REG 14,2 24890 1750306 /Users/stefan/Software/apache-tomcat-5.5.20/webapps/middletier/WEB-INF/lib/wicket-spring-1.2.5.jar
      java 9529 stefan 29r REG 14,2 486664 1750274 /Users/stefan/Software/apache-tomcat-5.5.20/webapps/middletier/WEB-INF/lib/wicket-extensions-1.2.5.jar
      java 9529 stefan 30r REG 14,2 486664 1750274 /Users/stefan/Software/apache-tomcat-5.5.20/webapps/middletier/WEB-INF/lib/wicket-extensions-1.2.5.jar
      java 9529 stefan 31r REG 14,2 486664 1750274 /Users/stefan/Software/apache-tomcat-5.5.20/webapps/middletier/WEB-INF/lib/wicket-extensions-1.2.5.jar
      java 9529 stefan 32r REG 14,2 486664 1750274 /Users/stefan/Software/apache-tomcat-5.5.20/webapps/middletier/WEB-INF/lib/wicket-extensions-1.2.5.jar
      java 9529 stefan 33r REG 14,2 486664 1750274 /Users/stefan/Software/apache-tomcat-5.5.20/webapps/middletier/WEB-INF/lib/wicket-extensions-1.2.5.jar
      java 9529 stefan 34r REG 14,2 486664 1750274 /Users/stefan/Software/apache-tomcat-5.5.20/webapps/middletier/WEB-INF/lib/wicket-extensions-1.2.5.jar
      java 9529 stefan 35r REG 14,2 486664 1750274 /Users/stefan/Software/apache-tomcat-5.5.20/webapps/middletier/WEB-INF/lib/wicket-extensions-1.2.5.jar
      java 9529 stefan 36r REG 14,2 486664 1750274 /Users/stefan/Software/apache-tomcat-5.5.20/webapps/middletier/WEB-INF/lib/wicket-extensions-1.2.5.jar
      java 9529 stefan 37r REG 14,2 486664 1750274 /Users/stefan/Software/apache-tomcat-5.5.20/webapps/middletier/WEB-INF/lib/wicket-extensions-1.2.5.jar
      java 9529 stefan 38r REG 14,2 486664 1750274 /Users/stefan/Software/apache-tomcat-5.5.20/webapps/middletier/WEB-INF/lib/wicket-extensions-1.2.5.jar
      java 9529 stefan 39r REG 14,2 486664 1750274 /Users/stefan/Software/apache-tomcat-5.5.20/webapps/middletier/WEB-INF/lib/wicket-extensions-1.2.5.jar
      java 9529 stefan 40r REG 14,2 486664 1750274 /Users/stefan/Software/apache-tomcat-5.5.20/webapps/middletier/WEB-INF/lib/wicket-extensions-1.2.5.jar
      java 9529 stefan 41r REG 14,2 486664 1750274 /Users/stefan/Software/apache-tomcat-5.5.20/webapps/middletier/WEB-INF/lib/wicket-extensions-1.2.5.jar
      [...many lines like this deleted]

        Activity

        Hide
        Stefan Wille added a comment -

        I have just tested our application with Wicket 1.2.6 on a fresh install of OpenSuse 10.2 Linux, and it works fine - no more leaking. If somebody else has a problem with leaked file handles, please try a later Linux version. For me, the issue is solved. Thank you for helping.

        I used Sun JDK 1.5.0_11 and Tomcat 5.5.20 for this test.

        Show
        Stefan Wille added a comment - I have just tested our application with Wicket 1.2.6 on a fresh install of OpenSuse 10.2 Linux, and it works fine - no more leaking. If somebody else has a problem with leaked file handles, please try a later Linux version. For me, the issue is solved. Thank you for helping. I used Sun JDK 1.5.0_11 and Tomcat 5.5.20 for this test.
        Hide
        Stefan Wille added a comment -

        For Wicket 1.2.5, we've seen the problem both on Mac OS X and on Linux. With Wicket 1.2.6, the problem seems to be fixed on Mac OS X.
        We are going to try Wicket 1.2.6 with Linux next week.

        Show
        Stefan Wille added a comment - For Wicket 1.2.5, we've seen the problem both on Mac OS X and on Linux. With Wicket 1.2.6, the problem seems to be fixed on Mac OS X. We are going to try Wicket 1.2.6 with Linux next week.
        Hide
        Eelco Hillenius added a comment -

        The problems I've seen in the past with this also solely were with Linux.

        Show
        Eelco Hillenius added a comment - The problems I've seen in the past with this also solely were with Linux.
        Hide
        Stefan Wille added a comment -

        We think that the remaining issue is caused by Linux, not Wicket.

        We have tried Jetty 6.1.1 instead of Tomcat 5.5 on our Linux server, which didn't help.
        Then we tried Wicket 1.3 with wicket-examples and Jetty on two other Linux machines, and the problem seems to be gone.
        These two machines have a newer kernel and probably a newer glibc.

        We still see the number of open wicket jar files increase by about 15, but it also decreases to the starting point, although slowly.

        Next Monday we are going to try the 1.2.6 development branch with our application on the two new Linux machines.

        Show
        Stefan Wille added a comment - We think that the remaining issue is caused by Linux, not Wicket. We have tried Jetty 6.1.1 instead of Tomcat 5.5 on our Linux server, which didn't help. Then we tried Wicket 1.3 with wicket-examples and Jetty on two other Linux machines, and the problem seems to be gone. These two machines have a newer kernel and probably a newer glibc. We still see the number of open wicket jar files increase by about 15, but it also decreases to the starting point, although slowly. Next Monday we are going to try the 1.2.6 development branch with our application on the two new Linux machines.
        Hide
        Johan Compagner added a comment -

        That code IS the fix if we don't test if it is jar url connection and we do just what is in the else then we have a much much bigger problem.
        Because then we are not able to close the jar file. And then we will leak a file handle everytime we access lastModifiedTime() (in developement mode you will run out of handles very shortly)

        The only thing i can think of is that there are constantly new ResourceReferences/Resources created somehow to the same resouce (gif/css/js)
        because in the constructor we have to do once: urlconnection.getContentType() and getContentLenght()
        that will open the jar file. But this shouldn't happen becaues (package)resources are registered and loaded once.

        Show
        Johan Compagner added a comment - That code IS the fix if we don't test if it is jar url connection and we do just what is in the else then we have a much much bigger problem. Because then we are not able to close the jar file. And then we will leak a file handle everytime we access lastModifiedTime() (in developement mode you will run out of handles very shortly) The only thing i can think of is that there are constantly new ResourceReferences/Resources created somehow to the same resouce (gif/css/js) because in the constructor we have to do once: urlconnection.getContentType() and getContentLenght() that will open the jar file. But this shouldn't happen becaues (package)resources are registered and loaded once.
        Hide
        Jean-Baptiste Quenot added a comment -

        In UrlResourceStream#lastModifiedTime() have you tried to remove the whole code within the "if (urlConnection instanceof JarURLConnection)" statement to see if it helps? Also be careful with the finally statement that might open a connection to the JAR if (close == true).

        Show
        Jean-Baptiste Quenot added a comment - In UrlResourceStream#lastModifiedTime() have you tried to remove the whole code within the "if (urlConnection instanceof JarURLConnection)" statement to see if it helps? Also be careful with the finally statement that might open a connection to the JAR if (close == true).
        Hide
        Johan Compagner added a comment -

        If you could see what code triggers a file handle then we can look at it.
        But i am affraid that we can't really do more about it, it really seems to be an jvm issue. We do try to close everything we can.
        (but ofcourse maybe somewhere there is a place where we forget it...)\

        You always can use the 2 tomcat context properties:

        antiJARLocking and antiResourceLocking: http://tomcat.apache.org/tomcat-5.5-doc/config/context.html

        Show
        Johan Compagner added a comment - If you could see what code triggers a file handle then we can look at it. But i am affraid that we can't really do more about it, it really seems to be an jvm issue. We do try to close everything we can. (but ofcourse maybe somewhere there is a place where we forget it...)\ You always can use the 2 tomcat context properties: antiJARLocking and antiResourceLocking: http://tomcat.apache.org/tomcat-5.5-doc/config/context.html
        Hide
        Ingram Chen added a comment -

        We have tried with latest Tomcat 6.0.10 but still have file-handle-grows-issue. This cause our site crashes twice a month...

        Show
        Ingram Chen added a comment - We have tried with latest Tomcat 6.0.10 but still have file-handle-grows-issue. This cause our site crashes twice a month...
        Hide
        Johan Compagner added a comment -

        and it needs to load all the resources at least once. The question is now does it grow and grow when
        there are really no new resources accessed?

        Show
        Johan Compagner added a comment - and it needs to load all the resources at least once. The question is now does it grow and grow when there are really no new resources accessed?
        Hide
        Eelco Hillenius added a comment -

        Ensure it realy is Wicket though. For my day project we recently fixed a file leak bug in Jetty (6.0/ 6.1), so it is entirely possible it's the container.

        Show
        Eelco Hillenius added a comment - Ensure it realy is Wicket though. For my day project we recently fixed a file leak bug in Jetty (6.0/ 6.1), so it is entirely possible it's the container.
        Hide
        Stefan Wille added a comment -

        Wicket 1.3 has the same problem. We we were not able to use our application with 1.3 because of the changed package names, but we tried wicket-examples on our Linux server. We saw the same situation.

        We have disabled the browser cache. In wicket-examples, we opened the "Tree and TreeTable" example in the ajax demo. Then we clicked "Expand All Nodes". We got this:

        $ while [ true ]; do lsof | grep wicket | wc -l; sleep 1;done
        127
        127
        127
        127
        125
        185
        185
        185
        185
        ...

        The jump to 185 file handles happened when we clicked the link.

        The server has Java version "1.5.0_07", Linux 2.6.8, Tomcat 5.5.17.

        Can I do anything to help with this issue?

        Show
        Stefan Wille added a comment - Wicket 1.3 has the same problem. We we were not able to use our application with 1.3 because of the changed package names, but we tried wicket-examples on our Linux server. We saw the same situation. We have disabled the browser cache. In wicket-examples, we opened the "Tree and TreeTable" example in the ajax demo. Then we clicked "Expand All Nodes". We got this: $ while [ true ]; do lsof | grep wicket | wc -l; sleep 1;done 127 127 127 127 125 185 185 185 185 ... The jump to 185 file handles happened when we clicked the link. The server has Java version "1.5.0_07", Linux 2.6.8, Tomcat 5.5.17. Can I do anything to help with this issue?
        Hide
        Stefan Wille added a comment -

        Hi Johan, thank you. We have built Wicket from the branch 1.2.x in order to try your backport. The behaviour is the same as with the backport from 2.0 that we did ourselves - we still get more than 900 open wicket-extensions.jar files.

        Next we will try 1.3.

        Show
        Stefan Wille added a comment - Hi Johan, thank you. We have built Wicket from the branch 1.2.x in order to try your backport. The behaviour is the same as with the backport from 2.0 that we did ourselves - we still get more than 900 open wicket-extensions.jar files. Next we will try 1.3.
        Hide
        Johan Compagner added a comment -

        backported it for 1.2.6
        Dont know why you still are able to crash tomcat. with that fix
        do you still have that also with 1.3?

        Show
        Johan Compagner added a comment - backported it for 1.2.6 Dont know why you still are able to crash tomcat. with that fix do you still have that also with 1.3?
        Hide
        Stefan Wille added a comment -

        We have backported the file UrlResourceStream.java from the trunk. This version works perfectly on Mac OS X. On Linux, the problem is reduced, but we can still get Tomcat to the point that it produces errors. We would appreciate if somebody from the Wicket team could do a proper backport of the workaround from 1.3.

        I have attached our copy of UrlResourceStream.java in case somebody would like to use it.

        Show
        Stefan Wille added a comment - We have backported the file UrlResourceStream.java from the trunk. This version works perfectly on Mac OS X. On Linux, the problem is reduced, but we can still get Tomcat to the point that it produces errors. We would appreciate if somebody from the Wicket team could do a proper backport of the workaround from 1.3. I have attached our copy of UrlResourceStream.java in case somebody would like to use it.
        Hide
        Stefan Wille added a comment -

        Yes, I put my WebApplication in deployment mode. This is the code:

        @Override
        protected void init()

        { ... configure(DEPLOYMENT); }

        I just searched the mailing list. In this posting

        http://www.nabble.com/File-descriptor-leak-in-DEVELOPMENT-mode-tf2280488.html#a6340061

        Jean-Baptiste says the problem occurs not only in development but also in deployment mode.

        We are using Wicket 1.2.5, so a backport of the workaround to 1.2.x would be great.

        Show
        Stefan Wille added a comment - Yes, I put my WebApplication in deployment mode. This is the code: @Override protected void init() { ... configure(DEPLOYMENT); } I just searched the mailing list. In this posting http://www.nabble.com/File-descriptor-leak-in-DEVELOPMENT-mode-tf2280488.html#a6340061 Jean-Baptiste says the problem occurs not only in development but also in deployment mode. We are using Wicket 1.2.5, so a backport of the workaround to 1.2.x would be great.
        Hide
        Johan Compagner added a comment -

        that workaround is only in 1.3 i think
        We could backport this to 1.2.x if needed

        Show
        Johan Compagner added a comment - that workaround is only in 1.3 i think We could backport this to 1.2.x if needed
        Hide
        Jean-Baptiste Quenot added a comment -

        Did you ensure to deploy your application in deployment mode? Did you have a look at the mailing-list archives? This problem has been reported numerous times, and it's a bug in the JVM. But a workaround exists in Wicket.

        Show
        Jean-Baptiste Quenot added a comment - Did you ensure to deploy your application in deployment mode? Did you have a look at the mailing-list archives? This problem has been reported numerous times, and it's a bug in the JVM. But a workaround exists in Wicket.

          People

          • Assignee:
            Johan Compagner
            Reporter:
            Stefan Wille
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development