OpenEJB
  1. OpenEJB
  2. OPENEJB-1955

TomEE 1.5.1 SNAPSHOT (and CDI beans) running slow on my production server

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Invalid
    • Affects Version/s: (not version related)
    • Fix Version/s: (not version related)
    • Component/s: container system
    • Environment:
      Windows Server 2003, 32-bit, PrimeFaces 3.5 SNAPSHOT, PrimeFaces Push (Atmosphere/websockets) configured;

      Internet connection is powered by Verizon Wireless 4G

      Description

      I was working with Romain and Mark Struberg on this one, and decided to open an issue, so I could attach files and continue discussion here.

      My TomEE/CDI-managed-beans web application is running really slow on production server (Windows Server 2003 32-bit Verizon Wireless 4G internet connection); runs faster on Windows Server 2008 64bit (cablemodem internet connection).

      Currently in production, the Glassfish 3.1.2.2 and JSF-managed-beans version of the web application is running much faster than the TomEE/CDI-managed-beans version of the web application.

      Per Romain and Mark, I ran jvisualvm to provide some benchmarks.

      Please review attached files and confirm and advise.

      1. jvisualvm_tomee.html
        9.00 MB
        Howard W. Smith, Jr.
      2. jvisualvm_tomee.nps
        120 kB
        Howard W. Smith, Jr.
      3. web.xml
        13 kB
        Howard W. Smith, Jr.
      4. jvisualvm_tomee_excel_smaller.csv
        2.55 MB
        Howard W. Smith, Jr.
      5. jvisualvm_tomee_smaller.csv
        2.53 MB
        Howard W. Smith, Jr.
      6. jvisualvm_tomee_smaller.html
        4.22 MB
        Howard W. Smith, Jr.
      7. jvisualvm.csv
        1.54 MB
        Howard W. Smith, Jr.
      8. jvisualvm.html
        2.65 MB
        Howard W. Smith, Jr.
      9. jvisualvm.nps
        45 kB
        Howard W. Smith, Jr.
      10. catalina.2012-11-27.log
        1.01 MB
        Howard W. Smith, Jr.
      11. jvisualvm_20121127.csv
        4.16 MB
        Howard W. Smith, Jr.
      12. jvisualvm_20121127.html
        7.20 MB
        Howard W. Smith, Jr.
      13. jvisualvm_20121127.nps
        103 kB
        Howard W. Smith, Jr.
      14. 20121128_profileLogin1.jpg
        121 kB
        Howard W. Smith, Jr.
      15. 20121128_profileLogin2.jpg
        118 kB
        Howard W. Smith, Jr.
      16. 20121128_profileLogin3.jpg
        124 kB
        Howard W. Smith, Jr.
      17. 20121128_profileLogin1_getOrderNumberList.jpg
        137 kB
        Howard W. Smith, Jr.
      18. 20121128_profileLogin2_getOrderNumberList.jpg
        133 kB
        Howard W. Smith, Jr.
      19. 20121128_profileLogin3_getOrderNumberList.jpg
        132 kB
        Howard W. Smith, Jr.
      20. 20121128_profileLogin_filterByDynamicSQL.jpg
        161 kB
        Howard W. Smith, Jr.
      21. 20121128_profileLogin_filterByNamedQuery.jpg
        167 kB
        Howard W. Smith, Jr.
      22. 20121128_profileLogin_initOrdersController1.jpg
        139 kB
        Howard W. Smith, Jr.
      23. 20121128_profileLogin_initOrdersController2.jpg
        149 kB
        Howard W. Smith, Jr.
      24. 20121128_profileLogin_initOrdersController3.jpg
        142 kB
        Howard W. Smith, Jr.

        Activity

        Hide
        Howard W. Smith, Jr. added a comment -

        Of course, i have been successfully using TomEE ever since i opened this JIRA, and I was able to resolve this issue via the following:

        1. optimize web app performance (replace JSF rendered="#

        {EL}" with ui:include src="#{EL}

        ", replace dynamic SQL queries with named queries, etc...

        2. server has been upgraded to Windows Server 2008 64-bit 32GB
        3. non-wireless internet connection

        Please close this issue or somehow mark as 'developer issue'.

        Show
        Howard W. Smith, Jr. added a comment - Of course, i have been successfully using TomEE ever since i opened this JIRA, and I was able to resolve this issue via the following: 1. optimize web app performance (replace JSF rendered="# {EL}" with ui:include src="#{EL} ", replace dynamic SQL queries with named queries, etc... 2. server has been upgraded to Windows Server 2008 64-bit 32GB 3. non-wireless internet connection Please close this issue or somehow mark as 'developer issue'.
        Hide
        Howard W. Smith, Jr. added a comment -

        Today (or this evening), I replaced the use of rendered="#

        {EL expression}" with ui:include src="#{EL expression}

        ", and created new/additional pages for the rendered="..." content. That seems to have made TomEE/CDI perform better on the production machine.

        I pushed TomEE/CDI to production machine. I hope it will go well 'today' in production as a debut.

        Show
        Howard W. Smith, Jr. added a comment - Today (or this evening), I replaced the use of rendered="# {EL expression}" with ui:include src="#{EL expression} ", and created new/additional pages for the rendered="..." content. That seems to have made TomEE/CDI perform better on the production machine. I pushed TomEE/CDI to production machine. I hope it will go well 'today' in production as a debut.
        Hide
        Howard W. Smith, Jr. added a comment -

        It's no longer non-jta. It's using persistence unit (mcmsPU), which is the JTA datasource. This all happened after I installed latest SNAPSHOT (12.03.2012) that I downloaded....onto production server.

        Show
        Howard W. Smith, Jr. added a comment - It's no longer non-jta. It's using persistence unit (mcmsPU), which is the JTA datasource. This all happened after I installed latest SNAPSHOT (12.03.2012) that I downloaded....onto production server.
        Hide
        Romain Manni-Bucau added a comment -

        Not sure i follow you since with this persistence.xml you datasource is jta. eclipselinks probably optimize some request + cache results but you are not in resource_local mode

        Show
        Romain Manni-Bucau added a comment - Not sure i follow you since with this persistence.xml you datasource is jta. eclipselinks probably optimize some request + cache results but you are not in resource_local mode
        Hide
        Howard W. Smith, Jr. added a comment -

        My persistence.xml looks like below. I thought I saw somewhere that I need to keep jta and non-jta in persistence.xml. Please confirm.

        <?xml version="1.0" encoding="UTF-8"?>
        <persistence version="2.0"
        xmlns="http://java.sun.com/xml/ns/persistence"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd">
        <persistence-unit name="mcmsPU" transaction-type="JTA">
        <provider>org.eclipse.persistence.jpa.PersistenceProvider</provider>
        <jta-data-source>jdbc/mcmsJta</jta-data-source>
        <non-jta-data-source>jdbc/mcmsNonJta</non-jta-data-source>
        <exclude-unlisted-classes>false</exclude-unlisted-classes>
        <properties>
        <property name="eclipselink.target-database"
        value="org.eclipse.persistence.platform.database.DerbyPlatform"/>
        <property name="eclipselink.jdbc.cache-statements" value="true" />
        <property name="eclipselink.jdbc.cache-statements.size" value="100" />
        <property name="eclipselink.logging.parameters" value="false" />
        </properties>
        </persistence-unit>
        </persistence>

        Show
        Howard W. Smith, Jr. added a comment - My persistence.xml looks like below. I thought I saw somewhere that I need to keep jta and non-jta in persistence.xml. Please confirm. <?xml version="1.0" encoding="UTF-8"?> <persistence version="2.0" xmlns="http://java.sun.com/xml/ns/persistence" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/persistence http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd "> <persistence-unit name="mcmsPU" transaction-type="JTA"> <provider>org.eclipse.persistence.jpa.PersistenceProvider</provider> <jta-data-source>jdbc/mcmsJta</jta-data-source> <non-jta-data-source>jdbc/mcmsNonJta</non-jta-data-source> <exclude-unlisted-classes>false</exclude-unlisted-classes> <properties> <property name="eclipselink.target-database" value="org.eclipse.persistence.platform.database.DerbyPlatform"/> <property name="eclipselink.jdbc.cache-statements" value="true" /> <property name="eclipselink.jdbc.cache-statements.size" value="100" /> <property name="eclipselink.logging.parameters" value="false" /> </properties> </persistence-unit> </persistence>
        Hide
        Howard W. Smith, Jr. added a comment -

        Wow, my CDI app is running pretty fast via TomEE 1.5.1 SNAPSHOT on the production server. Testing right now. Only thing that concerns me is my datasource is non-JTA, maybe because I assigned eclipselink properties in persistence.xml.

        Please confirm.

        I'm about to install latest TomEE 1.5.1 SNAPSHOT (that I have on my test/dev server). Also, let me know if I should download latest TomEE 1.5.1 SNAPSHOT from the repository; i didn't check today, was tuning my app (web pages).

        Show
        Howard W. Smith, Jr. added a comment - Wow, my CDI app is running pretty fast via TomEE 1.5.1 SNAPSHOT on the production server. Testing right now. Only thing that concerns me is my datasource is non-JTA, maybe because I assigned eclipselink properties in persistence.xml. Please confirm. I'm about to install latest TomEE 1.5.1 SNAPSHOT (that I have on my test/dev server). Also, let me know if I should download latest TomEE 1.5.1 SNAPSHOT from the repository; i didn't check today, was tuning my app (web pages).
        Hide
        Howard W. Smith, Jr. added a comment -

        Sorry, never mind, I see instructions for eclipselink javaagent added as jvm option (to tomee).

        http://wiki.eclipse.org/EclipseLink/UserGuide/JPA/Advanced_JPA_Development/Performance/Weaving/Dynamic_Weaving

        My only question, they recommend javaagent (dynamic weaving), if # of classes (to weave) is a small number. I have 57 entity classes; do they consider that small or large number of classes to weave?

        I also see that someone else asked how to do this via NetBeans:

        http://stackoverflow.com/questions/10431819/how-to-apply-static-weaving-ant-task-with-eclipse-link-jpa-in-netbeans
        http://forums.netbeans.org/topic47988.html

        Show
        Howard W. Smith, Jr. added a comment - Sorry, never mind, I see instructions for eclipselink javaagent added as jvm option (to tomee). http://wiki.eclipse.org/EclipseLink/UserGuide/JPA/Advanced_JPA_Development/Performance/Weaving/Dynamic_Weaving My only question, they recommend javaagent (dynamic weaving), if # of classes (to weave) is a small number. I have 57 entity classes; do they consider that small or large number of classes to weave? I also see that someone else asked how to do this via NetBeans: http://stackoverflow.com/questions/10431819/how-to-apply-static-weaving-ant-task-with-eclipse-link-jpa-in-netbeans http://forums.netbeans.org/topic47988.html
        Hide
        Howard W. Smith, Jr. added a comment -

        Please tell me more or share a URL related to tomee eclipselink javaagent. Thanks.

        Show
        Howard W. Smith, Jr. added a comment - Please tell me more or share a URL related to tomee eclipselink javaagent. Thanks.
        Hide
        Howard W. Smith, Jr. added a comment -

        Interesting, thanks Romain. I was definitely considering that, after reading about it; this is definitely a friendly reminder for me as well.

        Show
        Howard W. Smith, Jr. added a comment - Interesting, thanks Romain. I was definitely considering that, after reading about it; this is definitely a friendly reminder for me as well.
        Hide
        Romain Manni-Bucau added a comment -

        Side note: maybe consider either to add to tomee eclipselink javaagent or enhance your entities at build time http://wiki.eclipse.org/EclipseLink/UserGuide/JPA/Advanced_JPA_Development/Performance/Weaving/Static_Weaving

        Show
        Romain Manni-Bucau added a comment - Side note: maybe consider either to add to tomee eclipselink javaagent or enhance your entities at build time http://wiki.eclipse.org/EclipseLink/UserGuide/JPA/Advanced_JPA_Development/Performance/Weaving/Static_Weaving
        Hide
        Howard W. Smith, Jr. added a comment -

        Oh, i updated JSF-managed-bean web app with these jpa performance tuning code changes as well, so it can run 'faster' (hopefully) on production server.

        Show
        Howard W. Smith, Jr. added a comment - Oh, i updated JSF-managed-bean web app with these jpa performance tuning code changes as well, so it can run 'faster' (hopefully) on production server.
        Hide
        Howard W. Smith, Jr. added a comment -

        Today (and this evening), I added query hints to jpa query calls, updated jdbc properties in persistence.xml (eclipselink performing tuning properties), and at least tested the code changes with my TomEE/CDI-managed-beans web app on my fast server (Windows Server 2008), and it seems just as fast as the Glassfish/JSF-managed beans web app (running on the slower server, Windows Server 2003).

        Next, I would like to creatively render my pages, conditionally, via bean properties instead of rendered="...". More than likely, i plan to use ui:include src="#

        {bean.page}

        ", but i need to create templates or copies of the xhtml pages.

        Romain recommended single transaction. Will note that here as a friendly reminder.

        Show
        Howard W. Smith, Jr. added a comment - Today (and this evening), I added query hints to jpa query calls, updated jdbc properties in persistence.xml (eclipselink performing tuning properties), and at least tested the code changes with my TomEE/CDI-managed-beans web app on my fast server (Windows Server 2008), and it seems just as fast as the Glassfish/JSF-managed beans web app (running on the slower server, Windows Server 2003). Next, I would like to creatively render my pages, conditionally, via bean properties instead of rendered="...". More than likely, i plan to use ui:include src="# {bean.page} ", but i need to create templates or copies of the xhtml pages. Romain recommended single transaction. Will note that here as a friendly reminder.
        Hide
        Howard W. Smith, Jr. added a comment -

        Actually, this can stay open. I still plan to eliminate use of rendered="..." in the xhtml pages, and I'd like to report back here any/all test results. Thanks again.

        Show
        Howard W. Smith, Jr. added a comment - Actually, this can stay open. I still plan to eliminate use of rendered="..." in the xhtml pages, and I'd like to report back here any/all test results. Thanks again.
        Hide
        Howard W. Smith, Jr. added a comment -

        Please close this issue. Glassfish 3.1.2.2 and JSF managed beans fulfilling requirements. For now, I'm going to maintain JSF managed beans and Glassfish 3.1.2.2 in production. I am considering using TomEE and PrimeFaces Push to push data to web clients. I think I heard that I may need a load balancer in place for that.

        Not giving up on TomEE quite yet, but I've already spent too much time on this effort.

        Thanks for all your help.

        Show
        Howard W. Smith, Jr. added a comment - Please close this issue. Glassfish 3.1.2.2 and JSF managed beans fulfilling requirements. For now, I'm going to maintain JSF managed beans and Glassfish 3.1.2.2 in production. I am considering using TomEE and PrimeFaces Push to push data to web clients. I think I heard that I may need a load balancer in place for that. Not giving up on TomEE quite yet, but I've already spent too much time on this effort. Thanks for all your help.
        Hide
        Howard W. Smith, Jr. added a comment -

        Jean-Louis,

        Please don't quote me on Glassfish caching right out the box. All I know, my JSF-managed-beans web app is running much faster on Glassfish 3.1.2.2 on Windows Server 2003 32bit 4GB RAM, faster than TomEE / CDI web app on Windows Server 2003 32bit 4GB RAM and Windows Server 2008 64bit 16GB RAM. For this reason, I cannot push my TomEE / CDI web app to production server 'yet' and expect endusers to appreciate the extreme slowness of responses from web app.

        Even though I put a lot of effort in optimizing the app so far, I am not seeing any performance improvement on Windows Server 2003 32bit platform. I think I see performance improvement on Windows Server 2008 64bit, but I am at a loss on optimizing web app to deploy it to TomEE on Windows Server 2003.

        I am not done yet with all of my planned optimizing yet.

        I'm already running JDK 1.7.07 on Windows Server 2003 32bit, but I see that there is another JDK or JRE update available, I think it's JDK 1.7.09, right? Maybe that addresses some issues related to Windows 32bit environments. Romain did tell me that app will only run as good as the JVM will allow it to run.

        Romain, honestly, I don't know if I'm waving the entities. I read something about weaving, but I don't know if that's what you're talking about. Okay about TIKA stuff; it's not a real big concern. It serializes itself on app startup only, and living on an CDI @ApplicationScoped bean.

        Show
        Howard W. Smith, Jr. added a comment - Jean-Louis, Please don't quote me on Glassfish caching right out the box. All I know, my JSF-managed-beans web app is running much faster on Glassfish 3.1.2.2 on Windows Server 2003 32bit 4GB RAM, faster than TomEE / CDI web app on Windows Server 2003 32bit 4GB RAM and Windows Server 2008 64bit 16GB RAM. For this reason, I cannot push my TomEE / CDI web app to production server 'yet' and expect endusers to appreciate the extreme slowness of responses from web app. Even though I put a lot of effort in optimizing the app so far, I am not seeing any performance improvement on Windows Server 2003 32bit platform. I think I see performance improvement on Windows Server 2008 64bit, but I am at a loss on optimizing web app to deploy it to TomEE on Windows Server 2003. I am not done yet with all of my planned optimizing yet. I'm already running JDK 1.7.07 on Windows Server 2003 32bit, but I see that there is another JDK or JRE update available, I think it's JDK 1.7.09, right? Maybe that addresses some issues related to Windows 32bit environments. Romain did tell me that app will only run as good as the JVM will allow it to run. Romain, honestly, I don't know if I'm waving the entities. I read something about weaving, but I don't know if that's what you're talking about. Okay about TIKA stuff; it's not a real big concern. It serializes itself on app startup only, and living on an CDI @ApplicationScoped bean.
        Hide
        Romain Manni-Bucau added a comment -

        do you wave the entities? i think glassfish maybe does it

        about TIKA stuff you can avoid serializing it using a transient field i think

        Show
        Romain Manni-Bucau added a comment - do you wave the entities? i think glassfish maybe does it about TIKA stuff you can avoid serializing it using a transient field i think
        Hide
        Jean-Louis MONTEIRO added a comment -

        Hi Howard,

        I read carefully all your messages and went into some VisualVM snapshots.
        Maybe I missed something but did not saw significant overhead in OpenEJB/TomEE.

        In some of them, I saw BVal.

        Anyway, yes the query cache can provide huge improvements cause you don't have the sql cpu time in the database itself, but you also don't have the time spent in network, extracting results, creating objects, etc.

        But, it has to be configured carefully and I'm my opinion should be primary on read only data (reference data to name but a few).
        On read/write data, the result is not that significant as you have o deal with transaction and synchronize accesses. Moreover, in a clustered environment, you have to set up a distributed cache which is really painful to configure and which provide a significant overhead. In other words, it's sometimes worse that without cache.

        With that big picture in mind, I'm not sure we have to provide default settings for cache in EclipseLink or Hibernate/OpenJPA btw.
        But, that could be interesting to put a page up on the site describing possible optimizations per JPA Provider. WDYT?
        I'm happy to create the page and let you fill it with EclipseLink specific configurations.

        Are you sure that's the only difference between glassfish and TomEE?
        That seems strange to me Glassfish is caching queries out of the box.

        Anyway, a big thanks for your investigations and time.

        Show
        Jean-Louis MONTEIRO added a comment - Hi Howard, I read carefully all your messages and went into some VisualVM snapshots. Maybe I missed something but did not saw significant overhead in OpenEJB/TomEE. In some of them, I saw BVal. Anyway, yes the query cache can provide huge improvements cause you don't have the sql cpu time in the database itself, but you also don't have the time spent in network, extracting results, creating objects, etc. But, it has to be configured carefully and I'm my opinion should be primary on read only data (reference data to name but a few). On read/write data, the result is not that significant as you have o deal with transaction and synchronize accesses. Moreover, in a clustered environment, you have to set up a distributed cache which is really painful to configure and which provide a significant overhead. In other words, it's sometimes worse that without cache. With that big picture in mind, I'm not sure we have to provide default settings for cache in EclipseLink or Hibernate/OpenJPA btw. But, that could be interesting to put a page up on the site describing possible optimizations per JPA Provider. WDYT? I'm happy to create the page and let you fill it with EclipseLink specific configurations. Are you sure that's the only difference between glassfish and TomEE? That seems strange to me Glassfish is caching queries out of the box. Anyway, a big thanks for your investigations and time.
        Hide
        Howard W. Smith, Jr. added a comment -

        Since I still have Glassfish 3.1.2.2 and JSF managed beans web application running on production server, I like to compare speed and performance. I know EclipseLink comes bundled with Glassfish, so I really think Glassfish is running web apps with optimized parameters to EclipseLink, because queries are fast.

        I'm wonder if TomEE is leaving it up to developer to fully optimize JPA within apps.

        I know TomEE would love to gain Glassfish users. Maybe it is a good idea for TomEE to optimize as much as possible for developers, especially for those migrating from Glassfish.

        Show
        Howard W. Smith, Jr. added a comment - Since I still have Glassfish 3.1.2.2 and JSF managed beans web application running on production server, I like to compare speed and performance. I know EclipseLink comes bundled with Glassfish, so I really think Glassfish is running web apps with optimized parameters to EclipseLink, because queries are fast. I'm wonder if TomEE is leaving it up to developer to fully optimize JPA within apps. I know TomEE would love to gain Glassfish users. Maybe it is a good idea for TomEE to optimize as much as possible for developers, especially for those migrating from Glassfish.
        Hide
        Howard W. Smith, Jr. added a comment -

        Seems best for me to add to my CDI @ApplicationScoped bean, since there is no need for each user to own their own copy of FileHandler.

        Show
        Howard W. Smith, Jr. added a comment - Seems best for me to add to my CDI @ApplicationScoped bean, since there is no need for each user to own their own copy of FileHandler.
        Hide
        Howard W. Smith, Jr. added a comment -

        Interesting. Even though FileHandler is defined as following on CDI @SessionScoped pf_OrdersController:

        private FileHandler fileHandler = null;

        OWB/TomEE initializes this class, and FileHandler's,

        Detector DETECTOR = null;

        gets instantiated, since TIKA's Detector class implements Serializable.

        I need to make FileHandler CDI @SessionScoped and use @Inject.

        Show
        Howard W. Smith, Jr. added a comment - Interesting. Even though FileHandler is defined as following on CDI @SessionScoped pf_OrdersController: private FileHandler fileHandler = null; OWB/TomEE initializes this class, and FileHandler's, Detector DETECTOR = null; gets instantiated, since TIKA's Detector class implements Serializable. I need to make FileHandler CDI @SessionScoped and use @Inject.
        Hide
        Howard W. Smith, Jr. added a comment -

        Since I'm using TIKA library, the TIKA's Detector class is added as a private member of my FileHandler class. Below is the definition of the class:

        package org.apache.tika.detect;

        import java.io.IOException;
        import java.io.InputStream;
        import java.io.Serializable;
        import org.apache.tika.metadata.Metadata;
        import org.apache.tika.mime.MediaType;

        public interface Detector extends Serializable

        { public MediaType detect(InputStream in, Metadata mtdt) throws IOException; }

        Since this extends Serializable, evidently, this is the reason why TomEE resource logic is called.

        I didn't plan on keeping TIKA in my app 'forever', but for now, I'll lazily instantiate my FileHandler class.

        Show
        Howard W. Smith, Jr. added a comment - Since I'm using TIKA library, the TIKA's Detector class is added as a private member of my FileHandler class. Below is the definition of the class: package org.apache.tika.detect; import java.io.IOException; import java.io.InputStream; import java.io.Serializable; import org.apache.tika.metadata.Metadata; import org.apache.tika.mime.MediaType; public interface Detector extends Serializable { public MediaType detect(InputStream in, Metadata mtdt) throws IOException; } Since this extends Serializable, evidently, this is the reason why TomEE resource logic is called. I didn't plan on keeping TIKA in my app 'forever', but for now, I'll lazily instantiate my FileHandler class.
        Hide
        Howard W. Smith, Jr. added a comment -

        I just attached the following:

        20121128_profileLogin_initOrdersController1.jpg
        20121128_profileLogin_initOrdersController2.jpg
        20121128_profileLogin_initOrdersController3.jpg

        These 3 screen captures show the following:

        1. pf_OrdersController @PostConstruct initializes a POJO called FileHandler, which uses TIKA for MimeTypes.

        2. This reference to TIKA for MimeTypes seems to call TomEE/container's resources logic, and significantly slows down initialization/injection of pf_OrdersController.

        3. Evidently, I need to 'lazily' initialize my utility class, FileHandler, since endusers are not using/accessing FileHandler logic much at all!

        Show
        Howard W. Smith, Jr. added a comment - I just attached the following: 20121128_profileLogin_initOrdersController1.jpg 20121128_profileLogin_initOrdersController2.jpg 20121128_profileLogin_initOrdersController3.jpg These 3 screen captures show the following: 1. pf_OrdersController @PostConstruct initializes a POJO called FileHandler, which uses TIKA for MimeTypes. 2. This reference to TIKA for MimeTypes seems to call TomEE/container's resources logic, and significantly slows down initialization/injection of pf_OrdersController. 3. Evidently, I need to 'lazily' initialize my utility class, FileHandler, since endusers are not using/accessing FileHandler logic much at all!
        Hide
        Howard W. Smith, Jr. added a comment -

        Okay, well, i'm really impressed with some results that I'm seeing already via a few setHint() I just added to 2 separate queries.

        I did the following:

        OrdersFacade.filterBy()
        1. this retrieves data selected by user (usually current date)
        2. this is the most commonly-used (or popular) query
        3. added the following to this (dynamic sql) query:

        .setHint("eclipselink.query-results-cache", "true");

        OrdersFacade.findAllConfirmed()
        1. retrieves data for the entire year, based on the date range selected by user (same date range passed to OrdersFacade.filterBy(), above)
        2. this is called everytime OrdersFacade.filterBy(), above, is called
        3. in production, this query is dynamic SQL
        4. today, I changed this to be a namedQuery
        5. with the code changes below, bval is NOT called!!!
        6. added the following to this query:

        q = getEntityManager().createNamedQuery(namedQuery);
        q.setParameter("from", dateFrom, TemporalType.TIMESTAMP)
        .setParameter("to", dateTo, TemporalType.TIMESTAMP)
        .setHint("eclipselink.query-results-cache", "true")
        .setHint("eclipselink.read-only", "true");

        Remember you told me to do the 'warmup'? I assume that can be the first retrieve, that can be cached, or at least the query results can be cached. So, I was just testing all of the code changes (above), and I can tell that the cache is being used. Here are the test results:

        1. WARMUP: login to app and findAllConfirmed() for the first time: took 5,599 ms

        2. Selected a FROM date, 11/19/2012, and findAllConfirmed() took 191 ms

        3. Selected TO date, 11/23/2012 (which results in larger result set), and findAllConfirmed() took 102 ms

        4. Selected a row on datatable and clicked VIEW, and then clicked Browse to return to page with dataTable, and findAllConfirmed() took 100 ms

        I may need to add the following code changes to many other queries that I've defined in the app. Of course, I have to be careful to make sure that I don't add 'read-only' query hint everywhere, but I think it is safe to add read-only query hint to findAllConfirmed().

        Show
        Howard W. Smith, Jr. added a comment - Okay, well, i'm really impressed with some results that I'm seeing already via a few setHint() I just added to 2 separate queries. I did the following: OrdersFacade.filterBy() 1. this retrieves data selected by user (usually current date) 2. this is the most commonly-used (or popular) query 3. added the following to this (dynamic sql) query: .setHint("eclipselink.query-results-cache", "true"); OrdersFacade.findAllConfirmed() 1. retrieves data for the entire year, based on the date range selected by user (same date range passed to OrdersFacade.filterBy(), above) 2. this is called everytime OrdersFacade.filterBy(), above, is called 3. in production, this query is dynamic SQL 4. today, I changed this to be a namedQuery 5. with the code changes below, bval is NOT called!!! 6. added the following to this query: q = getEntityManager().createNamedQuery(namedQuery); q.setParameter("from", dateFrom, TemporalType.TIMESTAMP) .setParameter("to", dateTo, TemporalType.TIMESTAMP) .setHint("eclipselink.query-results-cache", "true") .setHint("eclipselink.read-only", "true"); Remember you told me to do the 'warmup'? I assume that can be the first retrieve, that can be cached, or at least the query results can be cached. So, I was just testing all of the code changes (above), and I can tell that the cache is being used. Here are the test results: 1. WARMUP: login to app and findAllConfirmed() for the first time: took 5,599 ms 2. Selected a FROM date, 11/19/2012, and findAllConfirmed() took 191 ms 3. Selected TO date, 11/23/2012 (which results in larger result set), and findAllConfirmed() took 102 ms 4. Selected a row on datatable and clicked VIEW, and then clicked Browse to return to page with dataTable, and findAllConfirmed() took 100 ms I may need to add the following code changes to many other queries that I've defined in the app. Of course, I have to be careful to make sure that I don't add 'read-only' query hint everywhere, but I think it is safe to add read-only query hint to findAllConfirmed().
        Hide
        Howard W. Smith, Jr. added a comment -

        Attached the following screen captures from Java Visual VM:

        "20121128_profileLogin_filterByDynamicSQL.jpg"

        No geromino code called

        "20121128_profileLogin_filterByNamedQuery.jpg"

        geromino code called; i think geromino = bval

        Again, this testing is without validation-mode = NONE in persistence.xml

        Next, I would like to analyze getOrderNumberList().

        Show
        Howard W. Smith, Jr. added a comment - Attached the following screen captures from Java Visual VM: "20121128_profileLogin_filterByDynamicSQL.jpg" No geromino code called "20121128_profileLogin_filterByNamedQuery.jpg" geromino code called; i think geromino = bval Again, this testing is without validation-mode = NONE in persistence.xml Next, I would like to analyze getOrderNumberList().
        Hide
        Howard W. Smith, Jr. added a comment -

        Just wanted to share these 3 screen captures to show minimum detail related to getOrderNumberList()

        Show
        Howard W. Smith, Jr. added a comment - Just wanted to share these 3 screen captures to show minimum detail related to getOrderNumberList()
        Hide
        Howard W. Smith, Jr. added a comment -

        Also, I am currently 'not' using the validation-mode = NONE in persistence.xml. The 3 screen captures posted above reflect performance 'without' validation-mode = NONE.

        Show
        Howard W. Smith, Jr. added a comment - Also, I am currently 'not' using the validation-mode = NONE in persistence.xml. The 3 screen captures posted above reflect performance 'without' validation-mode = NONE.
        Hide
        Howard W. Smith, Jr. added a comment -

        Conclusion (based on the last 2 posts above),

        1. there is 'no' need to tune the JPA code that is called by pf_OrdersController.filterBy(). that code is running fast.

        2. there is a need to possibly optimize pf_OrdersController.getOrderNumberList()

        Show
        Howard W. Smith, Jr. added a comment - Conclusion (based on the last 2 posts above), 1. there is 'no' need to tune the JPA code that is called by pf_OrdersController.filterBy(). that code is running fast. 2. there is a need to possibly optimize pf_OrdersController.getOrderNumberList()
        Hide
        Howard W. Smith, Jr. added a comment -

        Please note the following:

        1. pf_OrdersController.filterBy() is the set of code selects data based on today's date by default

        2. pf_OrdersController.getOrderNumberList() is the set of code that will select a bunch of rows for dates (from 01/01/2012 to 12/31/2012, or FROM beginning of year to END of year), so that we can apply an ORDER # to each ORDER.

        Can implementation of #2 be improved? Yes, I can store the order # as a column on the database, and update that value based on the # of actual orders placed during the current year, but we're not here to discuss that now. We are here to discuss why is my code slower deploy to/as TomEE/CDI.

        Show
        Howard W. Smith, Jr. added a comment - Please note the following: 1. pf_OrdersController.filterBy() is the set of code selects data based on today's date by default 2. pf_OrdersController.getOrderNumberList() is the set of code that will select a bunch of rows for dates (from 01/01/2012 to 12/31/2012, or FROM beginning of year to END of year), so that we can apply an ORDER # to each ORDER. Can implementation of #2 be improved? Yes, I can store the order # as a column on the database, and update that value based on the # of actual orders placed during the current year, but we're not here to discuss that now. We are here to discuss why is my code slower deploy to/as TomEE/CDI.
        Hide
        Howard W. Smith, Jr. added a comment -

        20121128_profileLogin1.jpg

        Shows the performance of my original use of JPA, which includes dynamic SQL (SELECT * from table WHERE ...). Below is the code that was issued:

        q = getEntityManager().createQuery(queryStr);

        20121128_profileLogin2.jpg

        Shows performance after added eclipselink.read-only = true as query hint

        q = getEntityManager().createQuery(queryStr).setHint("eclipselink.read-only", "true");

        20121128_profileLogin3.jpg

        Shows performance of using named query

        q = getEntityManager().createNamedQuery("Orders.filterByDate");
        q.setParameter("from", dateFrom, TemporalType.TIMESTAMP).
        setParameter("to", dateTo, TemporalType.TIMESTAMP);

        Show
        Howard W. Smith, Jr. added a comment - 20121128_profileLogin1.jpg Shows the performance of my original use of JPA, which includes dynamic SQL (SELECT * from table WHERE ...). Below is the code that was issued: q = getEntityManager().createQuery(queryStr); 20121128_profileLogin2.jpg Shows performance after added eclipselink.read-only = true as query hint q = getEntityManager().createQuery(queryStr).setHint("eclipselink.read-only", "true"); 20121128_profileLogin3.jpg Shows performance of using named query q = getEntityManager().createNamedQuery("Orders.filterByDate"); q.setParameter("from", dateFrom, TemporalType.TIMESTAMP). setParameter("to", dateTo, TemporalType.TIMESTAMP);
        Hide
        Howard W. Smith, Jr. added a comment -

        Attached screen captures of profiling login 1, 2, and 3. will explain more in next comment.

        Show
        Howard W. Smith, Jr. added a comment - Attached screen captures of profiling login 1, 2, and 3. will explain more in next comment.
        Hide
        Howard W. Smith, Jr. added a comment -

        jvisualvm files from latest test; please note, LogSql was turned on, and now I don't know how to turn off. LogSql = false and removing LogSql from tomee.xml does not stop logging SQL

        Show
        Howard W. Smith, Jr. added a comment - jvisualvm files from latest test; please note, LogSql was turned on, and now I don't know how to turn off. LogSql = false and removing LogSql from tomee.xml does not stop logging SQL
        Hide
        Howard W. Smith, Jr. added a comment -

        Per Romain, I am attaching catalina log, which has times it took to execute SQL. Opened a few pages in the web app.

        You can search log for the following that I added in the log:

        ***

        Show
        Howard W. Smith, Jr. added a comment - Per Romain, I am attaching catalina log, which has times it took to execute SQL. Opened a few pages in the web app. You can search log for the following that I added in the log: ***
        Hide
        Howard W. Smith, Jr. added a comment -

        jvisualvm is not showing me how much time it takes to <init> pf_OrdersController; from what I see, pf_OrdersController is at the bottom of the stack, almost the last thing on that particular stack, and it appears as though the entire stack is taking 48,407 ms (approximately 48 seconds).

        I am glad to see/learn that all of the other members (other CDI @SessionScoped and @RequestScoped beans) are not initialized during pf_OrdersController <init>. I have confirmed that the other CDI beans are initialized when they are referenced by xhtml or another bean. That is a nice-to-know (or lesson-learned) for me.

        Show
        Howard W. Smith, Jr. added a comment - jvisualvm is not showing me how much time it takes to <init> pf_OrdersController; from what I see, pf_OrdersController is at the bottom of the stack, almost the last thing on that particular stack, and it appears as though the entire stack is taking 48,407 ms (approximately 48 seconds). I am glad to see/learn that all of the other members (other CDI @SessionScoped and @RequestScoped beans) are not initialized during pf_OrdersController <init>. I have confirmed that the other CDI beans are initialized when they are referenced by xhtml or another bean. That is a nice-to-know (or lesson-learned) for me.
        Hide
        Howard W. Smith, Jr. added a comment -

        Deployed latest app changes to production server (Windows 2003 Server 32bit 4GB RAM).

        jvisualvm output files attached only includes app startup, user login, and first page that is displayed to user after successful user login.

        In other words, this test is primarily testing to see how long it takes for pf_OrdersController to be initialized in/by the container. Please note, pf_OrdersController is CDI @SessionScoped bean that uses @Inject (and MyFaces CODI BeanManagerProvider) to inject other CDI @SessionScoped (and @RequestScoped) beans.

        also, please note that today's latest app changes was an attempt to avoid CDI cyclic references; that is why I used MyFaces CODI BeanManagerProvider to the best of my ability to accomplish this.

        The test result is still unsatisfactory; the web app is still very very (or 'too') slow to push this TomEE/CDI web app version to 'production.

        Of course, per this JIRA/issue, I am attempting to 'tune' the app, especially all CDI managed beans, so this TomEE/CDI web app version will perform 'better than' or similar to Glassfish3.1.2.2/JSF-managed-beans web app that is currently in production...on production server.

        Show
        Howard W. Smith, Jr. added a comment - Deployed latest app changes to production server (Windows 2003 Server 32bit 4GB RAM). jvisualvm output files attached only includes app startup, user login, and first page that is displayed to user after successful user login. In other words, this test is primarily testing to see how long it takes for pf_OrdersController to be initialized in/by the container. Please note, pf_OrdersController is CDI @SessionScoped bean that uses @Inject (and MyFaces CODI BeanManagerProvider) to inject other CDI @SessionScoped (and @RequestScoped) beans. also, please note that today's latest app changes was an attempt to avoid CDI cyclic references; that is why I used MyFaces CODI BeanManagerProvider to the best of my ability to accomplish this. The test result is still unsatisfactory; the web app is still very very (or 'too') slow to push this TomEE/CDI web app version to 'production. Of course, per this JIRA/issue, I am attempting to 'tune' the app, especially all CDI managed beans, so this TomEE/CDI web app version will perform 'better than' or similar to Glassfish3.1.2.2/JSF-managed-beans web app that is currently in production...on production server.
        Hide
        Howard W. Smith, Jr. added a comment -

        Since TomEE 1.5.1 (Plus) SNAPSHOT currently does not implement CDI 1.1 (as mentioned above), I decided to do the following:

        1) replaced all @Inject with @EJB for sessionFacade classes injected into CDI managed beans; Romain/OpenEJB says that it is a very little bit faster to use @EJB instead of @Inject

        2) Modified pf_ChargesController: removed reference to pf_OrderCostDetailsController

        3) Modified AddressType, EmailAddressType, MethodOfPayment, PhoneType, MealStopType controllers: request scoped instead of session scoped, and remove most of the 'controller' code, since these controllers only need the 'byName' and 'SelectOne' methods

        4) Modified many beans/controllers to use CODI BeanManagerProvider.getInstance().getContextualReference() to simulate FacesContext.getCurrentInstance().getExternalContext().getSessionMap().get() that was used in the JSF managed bean code/baseline version of the app, which is currently in production

        Need to deploy the app (with these changes) to production server and re-test, when production server is available

        Show
        Howard W. Smith, Jr. added a comment - Since TomEE 1.5.1 (Plus) SNAPSHOT currently does not implement CDI 1.1 (as mentioned above), I decided to do the following: 1) replaced all @Inject with @EJB for sessionFacade classes injected into CDI managed beans; Romain/OpenEJB says that it is a very little bit faster to use @EJB instead of @Inject 2) Modified pf_ChargesController: removed reference to pf_OrderCostDetailsController 3) Modified AddressType, EmailAddressType, MethodOfPayment, PhoneType, MealStopType controllers: request scoped instead of session scoped, and remove most of the 'controller' code, since these controllers only need the 'byName' and 'SelectOne' methods 4) Modified many beans/controllers to use CODI BeanManagerProvider.getInstance().getContextualReference() to simulate FacesContext.getCurrentInstance().getExternalContext().getSessionMap().get() that was used in the JSF managed bean code/baseline version of the app, which is currently in production Need to deploy the app (with these changes) to production server and re-test, when production server is available
        Hide
        Howard W. Smith, Jr. added a comment -

        Latest per email conversation with Romain:

        You mentioned:

        about your session management, you should let it be done by the container...or assume results

        My response:

        Do you agree that TomEE needs to tune it's performance when injecting BeanA and BeanB, when they are both injected into each other, and when BeanA has other beans that also inject BeanA into themselves?

        Romain's response:

        this kind of injections will be fixed in CDI 1.1 (not JavaEE 6)...and often means a design error....

        that's saiid with some scope it shouldn't hurt that much since only proxies are injected

        Show
        Howard W. Smith, Jr. added a comment - Latest per email conversation with Romain: You mentioned: about your session management, you should let it be done by the container...or assume results My response: Do you agree that TomEE needs to tune it's performance when injecting BeanA and BeanB, when they are both injected into each other, and when BeanA has other beans that also inject BeanA into themselves? Romain's response: this kind of injections will be fixed in CDI 1.1 (not JavaEE 6)...and often means a design error.... that's saiid with some scope it shouldn't hurt that much since only proxies are injected
        Hide
        Howard W. Smith, Jr. added a comment -

        Willing to try applicationscoped just for test. Thanks.

        Interesting point about ejb. TomEE ejb examples web page demonstrated using @inject to inject @stateless ejb. If you recommend using @ejb, then I will do that.

        Your response just reminded me that when I initially migrated all code from JSF managed beans to CDI managed beans, that I used @inject to inject beans that were instantiated via session#getAttribute. Orderscontroller injects ordercostdetailscontroller, and ordercostdetailscontroller injects Orders controller. So my CDI code is much different than JSF managed code. I may need to use CODI bean reference utility for those cases to avoid cyclic injection when both are injected. Agree?

        Just recognized that using CODI bean reference was already on my to-do list for today.

        Show
        Howard W. Smith, Jr. added a comment - Willing to try applicationscoped just for test. Thanks. Interesting point about ejb. TomEE ejb examples web page demonstrated using @inject to inject @stateless ejb. If you recommend using @ejb, then I will do that. Your response just reminded me that when I initially migrated all code from JSF managed beans to CDI managed beans, that I used @inject to inject beans that were instantiated via session#getAttribute. Orderscontroller injects ordercostdetailscontroller, and ordercostdetailscontroller injects Orders controller. So my CDI code is much different than JSF managed code. I may need to use CODI bean reference utility for those cases to avoid cyclic injection when both are injected. Agree? Just recognized that using CODI bean reference was already on my to-do list for today.
        Hide
        Romain Manni-Bucau added a comment -

        to check it re-test breaking your app a little bit (but just for the test) -> use applicationscoped for instance to be sure the bean is created only once

        btw wonder if you couldn't use an ejb to be sure you use a single transaction by method (maybe in one method you use several transactions so you "loose time")

        Show
        Romain Manni-Bucau added a comment - to check it re-test breaking your app a little bit (but just for the test) -> use applicationscoped for instance to be sure the bean is created only once btw wonder if you couldn't use an ejb to be sure you use a single transaction by method (maybe in one method you use several transactions so you "loose time")
        Hide
        Howard W. Smith, Jr. added a comment -

        pf_OrdersController is CDI @SessionScoped and injects many other CDI @SessionScoped beans and EJB @Stateless via @Inject. So, you can imagine that tomcat/tomee (as well as Windows Server 2003/2008) struggles to inject/instantiate pf_OrdersController. Agree?

        That probably explains the really long wait after login. After user succcessfully login, pf_OrdersController is injected, because an ORDERS page is displayed/rendered to/for enduser in the browser.

        Show
        Howard W. Smith, Jr. added a comment - pf_OrdersController is CDI @SessionScoped and injects many other CDI @SessionScoped beans and EJB @Stateless via @Inject. So, you can imagine that tomcat/tomee (as well as Windows Server 2003/2008) struggles to inject/instantiate pf_OrdersController. Agree? That probably explains the really long wait after login. After user succcessfully login, pf_OrdersController is injected, because an ORDERS page is displayed/rendered to/for enduser in the browser.
        Hide
        Howard W. Smith, Jr. added a comment -

        Don't think CDI? Smiling... hard 'not' to think CDI. pf_OrdersController is huge clump of source code, it works well and is created very fast as JSF managed bean in a 'JSF managed bean' container (Glassfish...smile).

        I need to learn how to properly use and develop CDI managed beans that will run well in a container that can handle CDI (TomEE).

        Show
        Howard W. Smith, Jr. added a comment - Don't think CDI? Smiling... hard 'not' to think CDI. pf_OrdersController is huge clump of source code, it works well and is created very fast as JSF managed bean in a 'JSF managed bean' container (Glassfish...smile). I need to learn how to properly use and develop CDI managed beans that will run well in a container that can handle CDI (TomEE).
        Hide
        Romain Manni-Bucau added a comment -

        Hard to say without the code and with your figures but think that s somewhere else right...dont think to cdi btw

        Show
        Romain Manni-Bucau added a comment - Hard to say without the code and with your figures but think that s somewhere else right...dont think to cdi btw
        Hide
        Howard W. Smith, Jr. added a comment -

        pf_OrdersController is 'huge' in size (or byte count, lines of code). That is the core of the web application. I couldn't even find pf_OrdersController in jvisualvm output; I'm glad you found it. Evidently, I was looking at the output incorrectly.

        I just navigated through some of the pages. I guess it is best to do one jvisualvm output file per page in JSF web app.

        Along with what Mark recommended about rendered=, i need to eliminate/minimize use of rendered=... and I need to move much of the (potentially-stateless) code in pf_OrdersController to @CDI RequestScoped or @Stateless.

        So, PrimeFaces Push (Atmosphere) is not the cause of this problem? FYI, PrimeFaces Push (Atmosphere) is working perfectly/flawlessly in TomEE/Windows Server 2003 environment; it pushes. You think it is injection of pf_OrdersController?

        Show
        Howard W. Smith, Jr. added a comment - pf_OrdersController is 'huge' in size (or byte count, lines of code). That is the core of the web application. I couldn't even find pf_OrdersController in jvisualvm output; I'm glad you found it. Evidently, I was looking at the output incorrectly. I just navigated through some of the pages. I guess it is best to do one jvisualvm output file per page in JSF web app. Along with what Mark recommended about rendered=, i need to eliminate/minimize use of rendered=... and I need to move much of the (potentially-stateless) code in pf_OrdersController to @CDI RequestScoped or @Stateless. So, PrimeFaces Push (Atmosphere) is not the cause of this problem? FYI, PrimeFaces Push (Atmosphere) is working perfectly/flawlessly in TomEE/Windows Server 2003 environment; it pushes. You think it is injection of pf_OrdersController?
        Hide
        Romain Manni-Bucau added a comment -

        Your orders controller seems slow

        Bval too but seems that s the first parsing. How did you profile? How many times did you run your use case (seems not enough to get real figures)

        Show
        Romain Manni-Bucau added a comment - Your orders controller seems slow Bval too but seems that s the first parsing. How did you profile? How many times did you run your use case (seems not enough to get real figures)
        Hide
        Howard W. Smith, Jr. added a comment -

        smaller jvisualvm files

        Show
        Howard W. Smith, Jr. added a comment - smaller jvisualvm files
        Hide
        Howard W. Smith, Jr. added a comment -

        As requested, jvisualvm output files, also web.xml, so you can see how i have atmosphere (or primefaces push) configured for the web app.

        Show
        Howard W. Smith, Jr. added a comment - As requested, jvisualvm output files, also web.xml, so you can see how i have atmosphere (or primefaces push) configured for the web app.

          People

          • Assignee:
            Unassigned
            Reporter:
            Howard W. Smith, Jr.
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 612h
              612h
              Remaining:
              Remaining Estimate - 612h
              612h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development