Wookie
  1. Wookie
  2. WOOKIE-404

Server crashed after long wait, while reloading widget

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.14.0
    • Component/s: Server
    • Labels:
      None
    • Environment:
      Java vm 6.x, tomcat 6.0.36

      Description

      Server crashed after reloading a widget.

      Situation is as follows:

      1.-) Widget loaded in a portal. In my case Rave but I suppose it will do the same in whatever portal you try.

      2.-) Once the widget is loaded I leaved it overnight.

      3.-) I tried to reload the widget (by moving it). So the server initiated a reload.

      4.-) Stack trace as follows.

      Expected behavior is take over db connection and reconnect if neccesary.

      ------ Stack trace ----

      Estado HTTP 500 - La ejecución del Filtro lanzó una excepción

      type Informe de Excepción

      mensaje La ejecución del Filtro lanzó una excepción

      descripción El servidor encontró un error interno que hizo que no pudiera rellenar este requerimiento.

      excepción

      javax.servlet.ServletException: La ejecución del Filtro lanzó una excepción
      org.apache.wookie.server.MainFilter.doFilter(MainFilter.java:63)

      causa raíz

      java.lang.ExceptionInInitializerError
      org.apache.openjpa.lib.util.ConcreteClassGenerator.newInstance(ConcreteClassGenerator.java:132)
      org.apache.openjpa.lib.jdbc.ConfiguringConnectionDecorator.decorate(ConfiguringConnectionDecorator.java:107)
      org.apache.openjpa.lib.jdbc.DecoratingDataSource.decorate(DecoratingDataSource.java:118)
      org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:107)
      org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connectInternal(JDBCStoreManager.java:998)
      org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connect(JDBCStoreManager.java:983)
      org.apache.openjpa.jdbc.kernel.JDBCStoreManager.getConnection(JDBCStoreManager.java:286)
      org.apache.openjpa.jdbc.kernel.PreparedSQLStoreQuery$PreparedSQLExecutor.executeQuery(PreparedSQLStoreQuery.java:103)
      org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1003)
      org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:861)
      org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:792)
      org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:542)
      org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:288)
      org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:302)
      org.apache.openjpa.persistence.QueryImpl.getSingleResult(QueryImpl.java:326)
      org.apache.wookie.beans.jpa.JPAPersistenceManager.findWidgetInstanceByIdKey(JPAPersistenceManager.java:696)
      org.apache.wookie.server.LocalizedResourceFilter.getWidgetFromRequest(LocalizedResourceFilter.java:245)
      org.apache.wookie.server.LocalizedResourceFilter.doFilter(LocalizedResourceFilter.java:74)
      org.apache.wookie.server.MainFilter.doFilter(MainFilter.java:63)

      causa raíz

      java.lang.reflect.InvocationTargetException
      sun.reflect.GeneratedConstructorAccessor74.newInstance(Unknown Source)
      sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      java.lang.reflect.Constructor.newInstance(Constructor.java:532)
      org.apache.openjpa.lib.util.ConcreteClassGenerator.newInstance(ConcreteClassGenerator.java:130)
      org.apache.openjpa.lib.jdbc.ConfiguringConnectionDecorator.decorate(ConfiguringConnectionDecorator.java:107)
      org.apache.openjpa.lib.jdbc.DecoratingDataSource.decorate(DecoratingDataSource.java:118)
      org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:107)
      org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connectInternal(JDBCStoreManager.java:998)
      org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connect(JDBCStoreManager.java:983)
      org.apache.openjpa.jdbc.kernel.JDBCStoreManager.getConnection(JDBCStoreManager.java:286)
      org.apache.openjpa.jdbc.kernel.PreparedSQLStoreQuery$PreparedSQLExecutor.executeQuery(PreparedSQLStoreQuery.java:103)
      org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1003)
      org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:861)
      org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:792)
      org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:542)
      org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:288)
      org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:302)
      org.apache.openjpa.persistence.QueryImpl.getSingleResult(QueryImpl.java:326)
      org.apache.wookie.beans.jpa.JPAPersistenceManager.findWidgetInstanceByIdKey(JPAPersistenceManager.java:696)
      org.apache.wookie.server.LocalizedResourceFilter.getWidgetFromRequest(LocalizedResourceFilter.java:245)
      org.apache.wookie.server.LocalizedResourceFilter.doFilter(LocalizedResourceFilter.java:74)
      org.apache.wookie.server.MainFilter.doFilter(MainFilter.java:63)

      causa raíz

      com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 42.336.211 milliseconds ago. The last packet sent successfully to the server was 42.336.211 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.
      sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
      sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      java.lang.reflect.Constructor.newInstance(Constructor.java:532)
      com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
      com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
      com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3352)
      com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1971)
      com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
      com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)
      com.mysql.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:4997)
      org.apache.tomcat.dbcp.dbcp.DelegatingConnection.setAutoCommit(DelegatingConnection.java:371)
      org.apache.tomcat.dbcp.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.setAutoCommit(PoolingDataSource.java:328)
      org.apache.openjpa.lib.jdbc.DelegatingConnection.setAutoCommit(DelegatingConnection.java:220)
      org.apache.openjpa.lib.jdbc.DelegatingConnection.setAutoCommit(DelegatingConnection.java:220)
      org.apache.openjpa.lib.jdbc.ConfiguringConnectionDecorator$ConfiguringConnection.setAutoCommit(ConfiguringConnectionDecorator.java:132)
      org.apache.openjpa.lib.jdbc.ConfiguringConnectionDecorator$ConfiguringConnection.<init>(ConfiguringConnectionDecorator.java:126)
      org.apache.openjpa.lib.jdbc.ConfiguringConnectionDecorator$ConfiguringConnection_.<init>(Unknown Source)
      sun.reflect.GeneratedConstructorAccessor74.newInstance(Unknown Source)
      sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      java.lang.reflect.Constructor.newInstance(Constructor.java:532)
      org.apache.openjpa.lib.util.ConcreteClassGenerator.newInstance(ConcreteClassGenerator.java:130)
      org.apache.openjpa.lib.jdbc.ConfiguringConnectionDecorator.decorate(ConfiguringConnectionDecorator.java:107)
      org.apache.openjpa.lib.jdbc.DecoratingDataSource.decorate(DecoratingDataSource.java:118)
      org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:107)
      org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connectInternal(JDBCStoreManager.java:998)
      org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connect(JDBCStoreManager.java:983)
      org.apache.openjpa.jdbc.kernel.JDBCStoreManager.getConnection(JDBCStoreManager.java:286)
      org.apache.openjpa.jdbc.kernel.PreparedSQLStoreQuery$PreparedSQLExecutor.executeQuery(PreparedSQLStoreQuery.java:103)
      org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1003)
      org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:861)
      org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:792)
      org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:542)
      org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:288)
      org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:302)
      org.apache.openjpa.persistence.QueryImpl.getSingleResult(QueryImpl.java:326)
      org.apache.wookie.beans.jpa.JPAPersistenceManager.findWidgetInstanceByIdKey(JPAPersistenceManager.java:696)
      org.apache.wookie.server.LocalizedResourceFilter.getWidgetFromRequest(LocalizedResourceFilter.java:245)
      org.apache.wookie.server.LocalizedResourceFilter.doFilter(LocalizedResourceFilter.java:74)
      org.apache.wookie.server.MainFilter.doFilter(MainFilter.java:63)

      causa raíz

      java.net.SocketException: Broken pipe
      java.net.SocketOutputStream.socketWrite0(Native Method)
      java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
      java.net.SocketOutputStream.write(SocketOutputStream.java:153)
      java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
      java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
      com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3333)
      com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1971)
      com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
      com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)
      com.mysql.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:4997)
      org.apache.tomcat.dbcp.dbcp.DelegatingConnection.setAutoCommit(DelegatingConnection.java:371)
      org.apache.tomcat.dbcp.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.setAutoCommit(PoolingDataSource.java:328)
      org.apache.openjpa.lib.jdbc.DelegatingConnection.setAutoCommit(DelegatingConnection.java:220)
      org.apache.openjpa.lib.jdbc.DelegatingConnection.setAutoCommit(DelegatingConnection.java:220)
      org.apache.openjpa.lib.jdbc.ConfiguringConnectionDecorator$ConfiguringConnection.setAutoCommit(ConfiguringConnectionDecorator.java:132)
      org.apache.openjpa.lib.jdbc.ConfiguringConnectionDecorator$ConfiguringConnection.<init>(ConfiguringConnectionDecorator.java:126)
      org.apache.openjpa.lib.jdbc.ConfiguringConnectionDecorator$ConfiguringConnection_.<init>(Unknown Source)
      sun.reflect.GeneratedConstructorAccessor74.newInstance(Unknown Source)
      sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      java.lang.reflect.Constructor.newInstance(Constructor.java:532)
      org.apache.openjpa.lib.util.ConcreteClassGenerator.newInstance(ConcreteClassGenerator.java:130)
      org.apache.openjpa.lib.jdbc.ConfiguringConnectionDecorator.decorate(ConfiguringConnectionDecorator.java:107)
      org.apache.openjpa.lib.jdbc.DecoratingDataSource.decorate(DecoratingDataSource.java:118)
      org.apache.openjpa.lib.jdbc.DecoratingDataSource.getConnection(DecoratingDataSource.java:107)
      org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connectInternal(JDBCStoreManager.java:998)
      org.apache.openjpa.jdbc.kernel.JDBCStoreManager.connect(JDBCStoreManager.java:983)
      org.apache.openjpa.jdbc.kernel.JDBCStoreManager.getConnection(JDBCStoreManager.java:286)
      org.apache.openjpa.jdbc.kernel.PreparedSQLStoreQuery$PreparedSQLExecutor.executeQuery(PreparedSQLStoreQuery.java:103)
      org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:1003)
      org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:861)
      org.apache.openjpa.kernel.QueryImpl.execute(QueryImpl.java:792)
      org.apache.openjpa.kernel.DelegatingQuery.execute(DelegatingQuery.java:542)
      org.apache.openjpa.persistence.QueryImpl.execute(QueryImpl.java:288)
      org.apache.openjpa.persistence.QueryImpl.getResultList(QueryImpl.java:302)
      org.apache.openjpa.persistence.QueryImpl.getSingleResult(QueryImpl.java:326)
      org.apache.wookie.beans.jpa.JPAPersistenceManager.findWidgetInstanceByIdKey(JPAPersistenceManager.java:696)
      org.apache.wookie.server.LocalizedResourceFilter.getWidgetFromRequest(LocalizedResourceFilter.java:245)
      org.apache.wookie.server.LocalizedResourceFilter.doFilter(LocalizedResourceFilter.java:74)
      org.apache.wookie.server.MainFilter.doFilter(MainFilter.java:63)

      nota La traza completa de la causa de este error se encuentra en los archivos de diario de Apache Tomcat/6.0.36.
      Apache Tomcat/6.0.36

        Activity

        Hide
        Scott Wilson added a comment -

        Looks like a "dead connection" problem. There are some suggestions on this here:

        http://stackoverflow.com/questions/2077081/connection-with-mysql-is-being-aborted-automaticly-how-to-configure-connector-j

        Show
        Scott Wilson added a comment - Looks like a "dead connection" problem. There are some suggestions on this here: http://stackoverflow.com/questions/2077081/connection-with-mysql-is-being-aborted-automaticly-how-to-configure-connector-j
        Show
        Scott Wilson added a comment - Some more ideas here: http://stackoverflow.com/questions/5636195/dbcp-and-hibernate-on-spring-doesnt-re-open-dead-connections-why
        Hide
        Scott Wilson added a comment -

        OK, try this patch:

            1. Eclipse Workspace Patch 1.0
              #P wookie
              Index: src/org/apache/wookie/beans/jpa/JPAModule.java
              ===================================================================
            • src/org/apache/wookie/beans/jpa/JPAModule.java (revision 1449485)
              +++ src/org/apache/wookie/beans/jpa/JPAModule.java (working copy)
              @@ -96,6 +96,21 @@
              dataSource.setMaxIdle(80);
              dataSource.setInitialSize(5);
              dataSource.setMaxOpenPreparedStatements(0);
              +
              +
              + //
              + // Configure database connection to validate stale connections.
              + // The validation query to use is database specific
              + //
              + String validationQuery = "SELECT 1";
              + if (dbType.equals("derby")) validationQuery = "VALUES 1";
              + if (dbType.equals("oracle")) validationQuery = "SELECT 1 FROM DUAL";
              + if (dbType.equals("postgresql")) validationQuery = "select version();";
              + if (dbType.equals("hsqldb")) validationQuery = "select 1 from INFORMATION_SCHEMA.SYSTEM_USERS";
              + if (dbType.equals("db2")) validationQuery = "select 1 from sysibm.sysdummy1";
              + if (dbType.equals("mssql")) validationQuery = "sql select 1";
              + dataSource.setTestOnBorrow(true);
              + dataSource.setValidationQuery(validationQuery);

        // Set up connection pool
        GenericObjectPool pool = new GenericObjectPool();

        Show
        Scott Wilson added a comment - OK, try this patch: Eclipse Workspace Patch 1.0 #P wookie Index: src/org/apache/wookie/beans/jpa/JPAModule.java =================================================================== src/org/apache/wookie/beans/jpa/JPAModule.java (revision 1449485) +++ src/org/apache/wookie/beans/jpa/JPAModule.java (working copy) @@ -96,6 +96,21 @@ dataSource.setMaxIdle(80); dataSource.setInitialSize(5); dataSource.setMaxOpenPreparedStatements(0); + + + // + // Configure database connection to validate stale connections. + // The validation query to use is database specific + // + String validationQuery = "SELECT 1"; + if (dbType.equals("derby")) validationQuery = "VALUES 1"; + if (dbType.equals("oracle")) validationQuery = "SELECT 1 FROM DUAL"; + if (dbType.equals("postgresql")) validationQuery = "select version();"; + if (dbType.equals("hsqldb")) validationQuery = "select 1 from INFORMATION_SCHEMA.SYSTEM_USERS"; + if (dbType.equals("db2")) validationQuery = "select 1 from sysibm.sysdummy1"; + if (dbType.equals("mssql")) validationQuery = "sql select 1"; + dataSource.setTestOnBorrow(true); + dataSource.setValidationQuery(validationQuery); // Set up connection pool GenericObjectPool pool = new GenericObjectPool();
        Hide
        Gonzalo Aguilar added a comment -

        Perfect!

        Even restarting server worked! I will try to leave it for a long time a try again, but I think it will work.

        Show
        Gonzalo Aguilar added a comment - Perfect! Even restarting server worked! I will try to leave it for a long time a try again, but I think it will work.
        Hide
        Gonzalo Aguilar added a comment -

        I have to implement this on my servers...

        Thank you for the references.

        Show
        Gonzalo Aguilar added a comment - I have to implement this on my servers... Thank you for the references.
        Hide
        Scott Wilson added a comment -

        Fixed.

        Show
        Scott Wilson added a comment - Fixed.
        Hide
        Gonzalo Aguilar added a comment -

        I confirm. Fixed.
        Thank you again.

        Show
        Gonzalo Aguilar added a comment - I confirm. Fixed. Thank you again.

          People

          • Assignee:
            Unassigned
            Reporter:
            Gonzalo Aguilar
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development