Jetspeed 2
  1. Jetspeed 2
  2. JS2-210

Resolving all deployment classloader problems

    Details

      Description

      The current implementation of handling deployment, redeployment and undeployment of Portlet Applications (PAs) contains
      several workarounds to prevent file locking of jars and classes on Windows and to allow layout portlets to run under
      the Jetspeed-2 context.

      After almost a day debugging and testing I found the real cause of all the problems we try to fix:
      our own URLClassLoaders we create to be able to access the portlet and portlet resource bundles in the PA.

      As a test, I replaced our own URLClassLoaders with the contextClassLoader from the PA, retrieved during the init
      of the JetspeedServletContainer. And viola, no more problem!

      I'm going to drastically refactor the whole deployment implementation to be able to only use the PA contextClassLoaders.

      The autodeployment from the deploy folder will be changed to infuse the PA war (if needed) and move it to the webapps
      folder so the autodeployment of the application server will handle the real deployment.
      Thus: the undeployment from the deploy folder for PAs will be gone!

      Redeployment will still work as expected though.

      For undeployment, an new undeployment feature in the PAM Portlet will be added.
      Additionally, other PA management features like upload/start/stop/reload/ should be added soon too. We need a separate
      issue for that though.

      Decorators (and soon layouts too) will still be handled as is done right now (they don't involve classloaders).

      For deployment of layout portlets I want to make the following changes:

      • must be deployed as jar, not as war
      • can only contain classes, no resources
      • resources required by a layout portlet must be deployed separately with a decorators or layouts jar
      • will not be deployed as web app but the jar will be loaded with a URLClassloader with the portal classloader
        as parent: the portlets will be invoked through a JetspeedContainerServlet from the portal itself.
        These changes will also fix the outstanding problem with running Jetspeed-2 under a different context!
        (see JS2-172 and JS2-182)

      I'm going to need several days to perform this refactoring so if anyone has problems with this plan please do comment within the next few days!

        Activity

        Hide
        Peter Meier added a comment -

        Ate,
        When refactoring the deployment mechanism, will you consider auto deployment of PAs embedded in EAR files as well? If not, when will it be addressed?
        Peter.

        Show
        Peter Meier added a comment - Ate, When refactoring the deployment mechanism, will you consider auto deployment of PAs embedded in EAR files as well? If not, when will it be addressed? Peter.
        Hide
        Ate Douma added a comment -

        I'm not considering EAR files right now, and I'm not sure I (personally) will.
        The JetspeedDeploy tool can be used to infuse a WAR file before deployment and then you don't need
        the autodeployment feature of J2 itself anymore but just rely on the autodeployment features of
        your webserver.
        For EAR files you can use the same procedure on your WAR files before storing them in the EAR.
        Then simply deploy the EAR file as usual.

        Of course, if you still want to be able to use J2 for EAR deployment you/someone can write a DeployEnterpriseApplicationEventListener and plug it in as one of the deployment listeners of the
        DeploymentManager. That should be quite simple to do, certainly after my refactoring

        Show
        Ate Douma added a comment - I'm not considering EAR files right now, and I'm not sure I (personally) will. The JetspeedDeploy tool can be used to infuse a WAR file before deployment and then you don't need the autodeployment feature of J2 itself anymore but just rely on the autodeployment features of your webserver. For EAR files you can use the same procedure on your WAR files before storing them in the EAR. Then simply deploy the EAR file as usual. Of course, if you still want to be able to use J2 for EAR deployment you/someone can write a DeployEnterpriseApplicationEventListener and plug it in as one of the deployment listeners of the DeploymentManager. That should be quite simple to do, certainly after my refactoring
        Hide
        Ate Douma added a comment -

        I've created a new CVS branch, deployment-refactoring, in which I've committed my changes as of now.
        The current version is tested with deployment to Tomcat 5.0.28 and Tomcat 5.5.8 using JDK 1.4.2_04 for both.

        I've not added much comments in the code yet, nor committed with lots of messages as I wanted to get this into
        the branch ASAP.
        But, when (if) this branch is merged back into the HEAD branch I'll try to document it as much as possible.

        I'd like to invite all the developer team members as well as interested users to check this branch out
        and test it as much as possible.

        For sure it isn't complete/finished yet so I certainly expect issues to be reported

        I did run the tests though and, except for two, all run without failure:

        • Most tests in TestSimpleDeployment I've commented out for now as I haven't had time to write new tests
          conforming the new deployment.
        • The TestSSOComponent failed on me when using HSQLDB with an SQLException (FK constraint violation),
          but not with Oracle9i.
          I haven't run the test with HSQLDB on the current CVS HEAD but I expect it to fail there too
          as this problem isn't related to the changes in the branch.

        Anyone going to test this branch should be aware of the following:

        • Decorator deployment isn't in place yet (I discovered tonight I forgot to write it).
          I already thought out how to implement it and I will try to provide a first version tomorrow.
        • Portlet Application undeployment (local or now) is not in place either, and isn't going
          to work the old way anyhow.
          This will have to be implemented within the PAM portlets and/or manager servlet like Tomcat has
          itself (offline unregistration directly through the database is also an option).
        • The decorator portlet application is now (locally) deployed under context jetspeed-layouts.
          This means that all the layout fragments in the psml documents now are changed.
          Like: jetspeed::VelocityTwoColumns -> jetspeed-layouts::VelocityTwoColumns

        Finally, you now can deploy Jetspeed under a different context!
        You have to configure this by hand after deployment (I'll add a build configuration for it later):

        • rename jetspeed.xml
        • rename context path="/jetspeed" and docbase="jetspeed" inside jetspeed.xml
        • rename jetspeed.war
        • rename $TOMCAT/webapps/jetspeed
          Thats all.

        Regards, Ate

        Show
        Ate Douma added a comment - I've created a new CVS branch, deployment-refactoring, in which I've committed my changes as of now. The current version is tested with deployment to Tomcat 5.0.28 and Tomcat 5.5.8 using JDK 1.4.2_04 for both. I've not added much comments in the code yet, nor committed with lots of messages as I wanted to get this into the branch ASAP. But, when (if) this branch is merged back into the HEAD branch I'll try to document it as much as possible. I'd like to invite all the developer team members as well as interested users to check this branch out and test it as much as possible. For sure it isn't complete/finished yet so I certainly expect issues to be reported I did run the tests though and, except for two, all run without failure: Most tests in TestSimpleDeployment I've commented out for now as I haven't had time to write new tests conforming the new deployment. The TestSSOComponent failed on me when using HSQLDB with an SQLException (FK constraint violation), but not with Oracle9i. I haven't run the test with HSQLDB on the current CVS HEAD but I expect it to fail there too as this problem isn't related to the changes in the branch. Anyone going to test this branch should be aware of the following: Decorator deployment isn't in place yet (I discovered tonight I forgot to write it). I already thought out how to implement it and I will try to provide a first version tomorrow. Portlet Application undeployment (local or now) is not in place either, and isn't going to work the old way anyhow. This will have to be implemented within the PAM portlets and/or manager servlet like Tomcat has itself (offline unregistration directly through the database is also an option). The decorator portlet application is now (locally) deployed under context jetspeed-layouts. This means that all the layout fragments in the psml documents now are changed. Like: jetspeed::VelocityTwoColumns -> jetspeed-layouts::VelocityTwoColumns Finally, you now can deploy Jetspeed under a different context! You have to configure this by hand after deployment (I'll add a build configuration for it later): rename jetspeed.xml rename context path="/jetspeed" and docbase="jetspeed" inside jetspeed.xml rename jetspeed.war rename $TOMCAT/webapps/jetspeed Thats all. Regards, Ate
        Hide
        Ate Douma added a comment -

        A reply I've given to questions of David Sean Taylor on the dev mailing list:

        David Sean Taylor wrote:
        > A few quick questions:
        >
        > * why are the method names changing from register to start
        I guess you are referring to the PortletApplicationManagement interface?

        Register and start(PortletApplication) are two different functionalities now.
        The start method invokes the (Registry) register method if a PA hasn't
        been registered before or if its portlet.xml checksum has changed.

        Furthermore, it invokes the PortletFactory register method, indicating the
        web application has started and thereby signaling the portal can access/invoke
        the portlets from this PA.

        Maybe I should change the "start" and "stop" method names to "init" and "destroy".
        I have thought about that before but just forgot to do. I'm not stuck on method
        names though so if you prefer something else please be my guest.
        You can change it yourself.

        Which brings me to an important issue for the whole Team:
        don't think I see this as my branch!

        If you find bugs, see something clearly wrong or stupid, or need changes for
        whatever reason: please take action.

        I hope we can commit this branch very soon but I also know there is still
        work to do to bring back lost features.

        Although branches are very easy to create and certainly the best solution for
        creating and presenting an alternative solution, I hate merging when there are
        many conflicting changes. Getting this done ASAP helps reducing conflicting changes.

        So, please lets all try to decide ASAP

        • if this branch is going to be merged
        • what needs to be done before the merge

        Any help with that I'd appreciate very much.

        > * why don't we support the registerAtInit option in the deployTool anymore
        Because it is required now. This is the primary functionality, already introduced
        as optional feature by Randy, needed for the new deployment and Portlet container
        interaction. Only after a Portlet Application has signaled "start" to the
        PortletApplicationManager it becomes available to the Portal for usage.
        Without it, its "not available", so "start" at init really must be "true".

        Show
        Ate Douma added a comment - A reply I've given to questions of David Sean Taylor on the dev mailing list: David Sean Taylor wrote: > A few quick questions: > > * why are the method names changing from register to start I guess you are referring to the PortletApplicationManagement interface? Register and start(PortletApplication) are two different functionalities now. The start method invokes the (Registry) register method if a PA hasn't been registered before or if its portlet.xml checksum has changed. Furthermore, it invokes the PortletFactory register method, indicating the web application has started and thereby signaling the portal can access/invoke the portlets from this PA. Maybe I should change the "start" and "stop" method names to "init" and "destroy". I have thought about that before but just forgot to do. I'm not stuck on method names though so if you prefer something else please be my guest. You can change it yourself. Which brings me to an important issue for the whole Team: don't think I see this as my branch! If you find bugs, see something clearly wrong or stupid, or need changes for whatever reason: please take action. I hope we can commit this branch very soon but I also know there is still work to do to bring back lost features. Although branches are very easy to create and certainly the best solution for creating and presenting an alternative solution, I hate merging when there are many conflicting changes. Getting this done ASAP helps reducing conflicting changes. So, please lets all try to decide ASAP if this branch is going to be merged what needs to be done before the merge Any help with that I'd appreciate very much. > * why don't we support the registerAtInit option in the deployTool anymore Because it is required now. This is the primary functionality, already introduced as optional feature by Randy, needed for the new deployment and Portlet container interaction. Only after a Portlet Application has signaled "start" to the PortletApplicationManager it becomes available to the Portal for usage. Without it, its "not available", so "start" at init really must be "true".
        Hide
        Ate Douma added a comment -

        I've (re)implemented a very simple decorator deployer now.
        Undeployment is, like with PA deployment, not available yet.

        It works almost the same as the "old" version except it doesn't use the (InMemory)SimpleRegistry anymore.
        Redeployment is therefore a bit different.
        It now only undeploys an existing decorator with the same "id" and "decorates" property.
        If you redeploy a decorator with a different "decorates" value the previous won't be undeployed!

        Show
        Ate Douma added a comment - I've (re)implemented a very simple decorator deployer now. Undeployment is, like with PA deployment, not available yet. It works almost the same as the "old" version except it doesn't use the (InMemory)SimpleRegistry anymore. Redeployment is therefore a bit different. It now only undeploys an existing decorator with the same "id" and "decorates" property. If you redeploy a decorator with a different "decorates" value the previous won't be undeployed!
        Hide
        Ate Douma added a comment -

        I'll provide more information about the changes later but because I don't have time left for today or tomorrow
        I want to give the most important fact already now: JBoss (3.2.7) support.

        Everything works fine under JBoss, except for isUserInRole(rolename). After a short investigation I can say that
        our current JAAS security layer will not work out of the box with JBoss (and probably neither on several other
        JAAS supporting application servers). I will come back on this later. Login etc does work correctly though.

        Here are the instructions for getting this branch working on JBoss 3.2.7 (I also tested 4.0.1 shortly and it seems
        to work but I can't give it a 100% yet):

        build jetspeed-2 for Tomcat (5 or 5.5) with:
        maven allClean allBuild quickStart

        copy the contents of $TOMCAT_HOME/webapps/jetspeed into $JBOSS/server/default/deploy/jetspeed.war.

        remove all commons-logging-<version>.jar and log4j-<version>.jar from both jetspeed.war/lib as well as from
        all the war files inside jetspeed.war/WEB-INF/deploy:
        (jpetstore.war, jsf-demo.war, demo.war, pam.war, perl.war, php.war, security.war and struts-demo.war)
        These jars will give classloader problems otherwise because JBoss provides them already through a shared classloader.

        copy the contents of $TOMCAT/shared/lib into $JBOSS/server/default/lib (presuming $TOMCAT/shared/lib was empty before building).

        check $JBOSS/server/default/deploy/jbossweb-tomcat50.sar/server.xml:

        • element DefaultContext attribute crossContext="true" (which is the default)

        change $JBOSS/server/default/deploy/jbossweb-tomcat50.sar/META-INF/jboss-service.xml:

        • <atttribute name="Java2ClassLoadingCompliance">false</attribute>
        • <attribute name="UseJBossWebLoader">false</attribute>

        change $JBOSS/server/default/deploy/jetspeed.war/WEB-INF/assembly/jetspeed-spring.xml:

        • comment out entry "ApplicationServerManager" from bean with id "PortalServices"
        • comment out bean with id "org.apache.jetspeed.tools.pamanager.servletcontainer.ApplicationServerManager"

        copy jetspeed-ds.xml (or an adapted version like jetspeed-mysql-ds.xml) from the jetspeed-2 src folder portal/src/resource into $JBOSS/server/default/deploy.

        copy your jdbc-driver (as configured in jetspeed-ds.xml) to $JBOSS/server/default/lib
        (note: if you use the example hsqldb.jar setup of Jetspeed-2, you don't need to do this as JBoss provides it already)

        add the contents of jboss-login-config.xml from the jetspeed-2 src folder portal/src/resource to $JBOSS/server/default/conf/login-config.xml

        Happy testing

        Show
        Ate Douma added a comment - I'll provide more information about the changes later but because I don't have time left for today or tomorrow I want to give the most important fact already now: JBoss (3.2.7) support. Everything works fine under JBoss, except for isUserInRole(rolename). After a short investigation I can say that our current JAAS security layer will not work out of the box with JBoss (and probably neither on several other JAAS supporting application servers). I will come back on this later. Login etc does work correctly though. Here are the instructions for getting this branch working on JBoss 3.2.7 (I also tested 4.0.1 shortly and it seems to work but I can't give it a 100% yet): build jetspeed-2 for Tomcat (5 or 5.5) with: maven allClean allBuild quickStart copy the contents of $TOMCAT_HOME/webapps/jetspeed into $JBOSS/server/default/deploy/jetspeed.war. remove all commons-logging-<version>.jar and log4j-<version>.jar from both jetspeed.war/lib as well as from all the war files inside jetspeed.war/WEB-INF/deploy: (jpetstore.war, jsf-demo.war, demo.war, pam.war, perl.war, php.war, security.war and struts-demo.war) These jars will give classloader problems otherwise because JBoss provides them already through a shared classloader. copy the contents of $TOMCAT/shared/lib into $JBOSS/server/default/lib (presuming $TOMCAT/shared/lib was empty before building). check $JBOSS/server/default/deploy/jbossweb-tomcat50.sar/server.xml: element DefaultContext attribute crossContext="true" (which is the default) change $JBOSS/server/default/deploy/jbossweb-tomcat50.sar/META-INF/jboss-service.xml: <atttribute name="Java2ClassLoadingCompliance">false</attribute> <attribute name="UseJBossWebLoader">false</attribute> change $JBOSS/server/default/deploy/jetspeed.war/WEB-INF/assembly/jetspeed-spring.xml: comment out entry "ApplicationServerManager" from bean with id "PortalServices" comment out bean with id "org.apache.jetspeed.tools.pamanager.servletcontainer.ApplicationServerManager" copy jetspeed-ds.xml (or an adapted version like jetspeed-mysql-ds.xml) from the jetspeed-2 src folder portal/src/resource into $JBOSS/server/default/deploy. copy your jdbc-driver (as configured in jetspeed-ds.xml) to $JBOSS/server/default/lib (note: if you use the example hsqldb.jar setup of Jetspeed-2, you don't need to do this as JBoss provides it already) add the contents of jboss-login-config.xml from the jetspeed-2 src folder portal/src/resource to $JBOSS/server/default/conf/login-config.xml Happy testing
        Hide
        Ate Douma added a comment -

        Its done. The branch is merged with CVS HEAD.

        For JBoss support I have updated (and simplyfied) instructions:

        build jetspeed-2 for Tomcat (5 or 5.5) with:
        maven allClean allBuild quickStart

        copy the contents of $TOMCAT_HOME/webapps/jetspeed into $JBOSS/server/default/deploy/jetspeed.war.

        remove commons-logging-<version>.jar and log4j-<version>.jar from jetspeed.war/lib

        copy the contents of $TOMCAT/shared/lib into $JBOSS/server/default/lib (presuming $TOMCAT/shared/lib was empty before building).

        check $JBOSS/server/default/deploy/jbossweb-tomcat50.sar/server.xml:

        • element DefaultContext attribute crossContext="true" (which is the default)

        change $JBOSS/server/default/deploy/jbossweb-tomcat50.sar/META-INF/jboss-service.xml:

        • <atttribute name="Java2ClassLoadingCompliance">false</attribute>
        • <attribute name="UseJBossWebLoader">false</attribute>

        change $JBOSS/server/default/deploy/jetspeed.war/WEB-INF/assembly/jetspeed-spring.xml:

        • comment out entry "ApplicationServerManager" from bean with id "PortalServices"
        • comment out bean with id "org.apache.jetspeed.tools.pamanager.servletcontainer.ApplicationServerManager"
        • change constructor-arg index="4" value to "true" for bean with id "portletAppDeploymentListener"
          This will strip commons-logging and log4j jars from the war file on deployment as required on JBoss

        copy jetspeed-ds.xml (or an adapted version like jetspeed-mysql-ds.xml) from the jetspeed-2 src folder portal/src/resource into $JBOSS/server/default/deploy.

        copy your jdbc-driver (as configured in jetspeed-ds.xml) to $JBOSS/server/default/lib
        If you use using the example hdsqldb setup of Jetspeed-2:

        • remove $JBOSS/server/default/lib/hsqldb.jar
        • move the hsqldb<-version>.jar from $JBOSS/server/default/default/jetspeed.war/WEB-INF/lib into $JBOSS/server/default/lib

        add the contents of jboss-login-config.xml from the jetspeed-2 src folder portal/src/resource to $JBOSS/server/default/conf/login-config.xml

        Show
        Ate Douma added a comment - Its done. The branch is merged with CVS HEAD. For JBoss support I have updated (and simplyfied) instructions: build jetspeed-2 for Tomcat (5 or 5.5) with: maven allClean allBuild quickStart copy the contents of $TOMCAT_HOME/webapps/jetspeed into $JBOSS/server/default/deploy/jetspeed.war. remove commons-logging-<version>.jar and log4j-<version>.jar from jetspeed.war/lib copy the contents of $TOMCAT/shared/lib into $JBOSS/server/default/lib (presuming $TOMCAT/shared/lib was empty before building). check $JBOSS/server/default/deploy/jbossweb-tomcat50.sar/server.xml: element DefaultContext attribute crossContext="true" (which is the default) change $JBOSS/server/default/deploy/jbossweb-tomcat50.sar/META-INF/jboss-service.xml: <atttribute name="Java2ClassLoadingCompliance">false</attribute> <attribute name="UseJBossWebLoader">false</attribute> change $JBOSS/server/default/deploy/jetspeed.war/WEB-INF/assembly/jetspeed-spring.xml: comment out entry "ApplicationServerManager" from bean with id "PortalServices" comment out bean with id "org.apache.jetspeed.tools.pamanager.servletcontainer.ApplicationServerManager" change constructor-arg index="4" value to "true" for bean with id "portletAppDeploymentListener" This will strip commons-logging and log4j jars from the war file on deployment as required on JBoss copy jetspeed-ds.xml (or an adapted version like jetspeed-mysql-ds.xml) from the jetspeed-2 src folder portal/src/resource into $JBOSS/server/default/deploy. copy your jdbc-driver (as configured in jetspeed-ds.xml) to $JBOSS/server/default/lib If you use using the example hdsqldb setup of Jetspeed-2: remove $JBOSS/server/default/lib/hsqldb.jar move the hsqldb<-version>.jar from $JBOSS/server/default/default/jetspeed.war/WEB-INF/lib into $JBOSS/server/default/lib add the contents of jboss-login-config.xml from the jetspeed-2 src folder portal/src/resource to $JBOSS/server/default/conf/login-config.xml
        Hide
        Ate Douma added a comment -

        The new deployment implmentation has been tested and used without problems for some time now.

        Show
        Ate Douma added a comment - The new deployment implmentation has been tested and used without problems for some time now.

          People

          • Assignee:
            Ate Douma
            Reporter:
            Ate Douma
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development