Description
This follows the discussion in dev ML at https://s.apache.org/358zz
Trunk HEAD, R17 and R18 are affected. R16 has a much lesser issue that can be neglected.
2019-08-16 09:14:14,453 |main |ConfigXMLReader |I| controller loaded: 0.003s, 207 requests, 81 views in file:/C:/projectsASF/release18.12/plugins/scrum/webapp/scrum/WEB-INF/controller.xml 2019-08-16 09:17:16,061 |main |ServiceDispatcher |I| Registering dispatcher: solr 2019-08-16 09:24:10,541 |main |ConfigXMLReader |I| controller loaded: 0.004s, 207 requests, 81 views in file:/C:/projectsASF/release17.12/plugins/scrum/webapp/scrum/WEB-INF/controller.xml 2019-08-16 09:27:12,125 |main |ServiceDispatcher |I| Registering dispatcher: solr
3 minutes to register solr on my machine!
The much lesser issue on my machine and demo with R16:
2019-08-16 09:30:54,464 |0.0.0.0-startStop-1 |ConfigXMLReader |I| controller loaded: 0.004s, 207 requests, 81 views in file:/C:/projectsASF/release16.11/specialpurpose/scrum/webapp/scrum/WEB-INF/controller.xml 2019-08-16 09:31:55,069 |0.0.0.0-startStop-1 |SolrResourceLoader |W| Can't find (or read) directory to add to classloader: lib (resolved as: C:\projectsASF\release16.11\specialpurpose\solr\home\lib). 2019-08-16 09:31:55,239 |0.0.0.0-startStop-1 |ServiceDispatcher |I| Registering dispatcher: solr 2019-08-16 03:12:33,016 |0.0.0.0-startStop-1 |ConfigXMLReader |I| controller loaded: 0.005s, 207 requests, 81 views in file:/home/ofbizDemo/branch16.11/specialpurpose/scrum/webapp/scrum/WEB-INF/controller.xml 2019-08-16 03:12:33,655 |0.0.0.0-startStop-1 |SolrResourceLoader |W| Can't find (or read) directory to add to classloader: lib (resolved as: /home/ofbizDemo/branch16.11/specialpurpose/solr/home/lib). 2019-08-16 03:12:33,879 |0.0.0.0-startStop-1 |ServiceDispatcher |I| Registering dispatcher: solr
600ms is barely an issue. So I think we can agree it's OK there, anyway nobody never complained.
In trunk, R17 and R18, it's due to SolrResourceLoader.locateSolrHome() trying to locate Solr home dir by 1st doing a JNDI lookup, then a system property and eventually harcode it to "solr/"
JNDI lookups can be slow. But here we speak about minutes because in Solr there are several calls w/o caching.
I thought that by avoiding the JNDI lookup done in OFBizSolrContextFilter::init (through super.init(config); I'd fix the issue. So I wrote this
Index: config/solrconfig.properties =================================================================== --- config/solrconfig.properties (revision 1864966) +++ config/solrconfig.properties (working copy) @@ -50,3 +50,7 @@ solr.log.dir=runtime/logs/solr # Defines Solr log level solr.log.level=INFO + +# Defines Solr home directory +solr.solr.home=plugins/solr/home + Index: src/main/java/org/apache/ofbiz/solr/webapp/OFBizSolrContextFilter.java =================================================================== --- src/main/java/org/apache/ofbiz/solr/webapp/OFBizSolrContextFilter.java (revision 1864966) +++ src/main/java/org/apache/ofbiz/solr/webapp/OFBizSolrContextFilter.java (working copy) @@ -66,6 +66,7 @@ Properties props = System.getProperties(); props.setProperty("solr.log.dir", UtilProperties.getPropertyValue("solrconfig", "solr.log.dir", "runtime/logs/solr")); props.setProperty("solr.log.level", UtilProperties.getPropertyValue("solrconfig", "solr.log.level", "INFO")); + config.getServletContext().setAttribute("solr.solr.home", UtilProperties.getPropertyValue("solrconfig", "solr.solr.home", "plugins/solr")); super.init(config); }
to prevent the SolrResourceLoader.locateSolrHome() calls in SolrDispatchFilter::init
It worked but it's obviously not enough. It's then 1 vs 3 minutes before, JNDI timeout I guess. Here it is seen with <logger name="org.apache.solr" level="all"/> in log4j2.xml)
2019-08-16 12:03:39,402 |main |SolrDispatchFilter |I| ___ _ Welcome to Apache Solr? version 8.2.0 2019-08-16 12:03:39,402 |main |SolrDispatchFilter |I| / __| ___| |_ _ Starting in standalone mode on port null 2019-08-16 12:03:39,402 |main |SolrDispatchFilter |I| \__ \/ _ \ | '_| Install dir: null 2019-08-16 12:03:39,403 |main |SolrDispatchFilter |I| |___/\___/_|_| Start time: 2019-08-16T10:03:39.402Z 2019-08-16 12:03:39,403 |main |SolrDispatchFilter |I| Log level override, property solr.log.level=INFO 2019-08-16 12:03:39,412 |main |SolrResourceLoader |D| new SolrResourceLoader for directory: 'C:\projectsASF\ofbiz\plugins\solr\home' 2019-08-16 12:04:39,421 |main |SolrResourceLoader |D| No /solr/home in JNDI 2019-08-16 12:04:39,421 |main |SolrResourceLoader |I| solr home defaulted to 'solr/' (could not find system property or JNDI) 2019-08-16 12:04:39,428 |main |SolrXmlConfig |I| Loading container configuration from C:\projectsASF\ofbiz\plugins\solr\home\solr.xml
There are 3 JNDI lookups in Solr (at least with our use of Solr) that locate the Solr home dir, hence the 2 minutes before my change. The JNDI context nor result are cached. BTW I believe Solr could do better by using https://docs.oracle.com/javase/8/docs/api/javax/naming/InitialContext.html#doLookup-java.lang.String- in SolrResourceLoader.locateSolrHome() and caching but that's out of subject.
I then tried to find a way to Configure Solr Home with JNDI. I tried many ways but none worked so far. Among them:
- As a try: add an <env-entry> in solr component web.xml file
IIndex: web.xml =================================================================== --- web.xml (revision 1864966) +++ web.xml (working copy) @@ -20,6 +20,12 @@ <display-name>Apache OFBiz - Solr Component</display-name> <description>Solr Component of the Apache OFBiz Project</description> + <env-entry> + <env-entry-name>solr/home</env-entry-name> + <env-entry-value>C:\projectsASF\ofbiz\plugins\solr\home</env-entry-value> + <env-entry-type>java.lang.String</env-entry-type> + </env-entry> + <context-param> <param-name>entityDelegatorName</param-name> <param-value>default</param-value>
Not sure why it did not work
- I tried to set in jndi.properties (though I believe it's not for JNDI env but ressource), like
solr/home= C:\projectsASF\ofbiz\plugins\solr\home
- Used OFBIZ-9484 to grab the content of a context.xml file:
@@ -514,6 +517,23 @@ context.setDisplayName(appInfo.name); context.setPath(getWebappMountPoint(appInfo)); context.addLifecycleListener(new ContextConfig()); + + // adding webapp's META-INF/context.xml begin + String contextXmlFilePath = new StringBuilder().append("file:///").append(location).append("/").append(Constants.ApplicationContextXml).toString(); + URL contextXmlUrl = null; + try { + contextXmlUrl = FlexibleLocation.resolveLocation(contextXmlFilePath); + contextXmlFilePath = new StringBuilder().append(location).append("/").append(Constants.ApplicationContextXml).toString(); + File contextXmlFile = FileUtil.getFile(contextXmlFilePath); + if(contextXmlFile.exists() && contextXmlFile.isFile()) { + Debug.logInfo(contextXmlFilePath + " found and will be loaded.", module); + context.setConfigFile(contextXmlFilePath); + } + } catch (MalformedURLException e) { + Debug.logInfo(contextXmlFilePath+ " not found.", module); + } + // add webapp's META-INF/context.xml end + context.setJ2EEApplication("OFBiz"); context.setJ2EEServer("OFBiz Container"); context.setLoader(new WebappLoader(Thread.currentThread().getContextClassLoader()));
And following https://tomcat.apache.org/tomcat-9.0-doc/config/context.html#Environment_Entries, added
<Context> <Environment name="solr/home" value="C:\projectsASF\ofbiz\plugins\solr\home" type="java.lang.String" description="Solr home for JNDI"/> </Context>
But reading https://docs.oracle.com/javase/jndi/tutorial/beyond/env/update.html I'm unsure context.setConfigFile(contextXmlFilePath); does the work of loading this Environment
I don't see a way to set a short timeout on the JNDI lookup. A solution like that could be used. That would need to be in Solr...
Getting back to R16 for lucene and solr component could maybe be a way, but it's really bad then. Another option is to not use Solr if you don't need it
Just before sending this comment I looked into our archive and BAM! Just 6 years ago: point 2 of my comment: https://s.apache.org/358zz
If I miss something it's very well hidden, all ideas are welcome.