[SOLR-6675] Solr webapp deployment is very slow with <jmx/> in solrconfig.xml

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Won't Fix
• Affects Version/s: 4.7
• Component/s: SolrCloud
• Labels: performance

Environment

Linux Redhat 64bit

• Assignee: Unassigned
• Reporter: Forest Soup
• Created: 30/Oct/14 08:12
• Updated: 02/Oct/16 03:26
• Resolved: 02/Oct/16 03:26

Description

We have a SolrCloud with Solr version 4.7 with Tomcat 7. And our solr index(cores) are big(50~100G) each core.

When we start up tomcat, the solr webapp deployment is very slow. From tomcat's catalina log, every time it takes about 10 minutes to get deployed. After we analyzing java core dump, we notice it's because the loading process cannot finish until the MBean calculation for large index is done.

So we tried to remove the <jmx/> from solrconfig.xml, after that, the loading of solr webapp only take about 1 minute. So we can sure the MBean calculation for large index is the root cause.

Could you please point me if there is any async way to do statistic monitoring without <jmx/> in solrconfig.xml, or let it do calculation after the deployment? Thanks!

The callstack.png file in the attachment is the call stack of the long blocking thread which is doing statistics calculation.

The catalina log of tomcat:
INFO: Starting Servlet Engine: Apache Tomcat/7.0.54
Oct 13, 2014 2:00:29 AM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war
Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deployment of web application archive /opt/ibm/solrsearch/tomcat/webapps/solr.war has finished in 594,325 ms <---------------- Time taken for solr app Deployment is about 10 minutes -----------
Oct 13, 2014 2:10:23 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/manager
Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/manager has finished in 2,035 ms
Oct 13, 2014 2:10:26 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/examples
Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/examples has finished in 1,789 ms
Oct 13, 2014 2:10:27 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/docs
Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/docs has finished in 1,037 ms
Oct 13, 2014 2:10:28 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT
Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/ROOT has finished in 948 ms
Oct 13, 2014 2:10:29 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deploying web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager
Oct 13, 2014 2:10:30 AM org.apache.catalina.startup.HostConfig deployDirectory
INFO: Deployment of web application directory /opt/ibm/solrsearch/tomcat/webapps/host-manager has finished in 951 ms
Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8080"]
Oct 13, 2014 2:10:31 AM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Oct 13, 2014 2:10:31 AM org.apache.catalina.startup.Catalina start
INFO: Server startup in 601506 ms

Issue Links

• relates to: SOLR-6586 JmxMonitoredMap#getAttribute is not very efficient.
• relates to: SOLR-6747 Add an optional caching option as a workaround for SOLR-6586.