[SOLR-12294] System collection - Lazy loading mechanism not working for custom UpdateProcessors

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 7.3
• Fix Version/s: 7.4, 8.0
• Component/s: UpdateRequestProcessors
• Assignee: Noble Paul
• Reporter: Johannes Brucher
• Created: 30/Apr/18 14:39
• Updated: 08/Jun/19 15:11
• Resolved: 26/Jun/18 12:14

Description

Hi all,

I'm facing an issue regarding custom code inside a .system-collection and starting up a Solr Cloud cluster.

I thought, like its stated in the documentation, that in case using the .system collection custom code is lazy loaded, because it can happen that a collection that uses custom code is initialized before the .system collection is up and running.

I did all the necessary configuration and while debugging, I can see that the custom code is wrapped via a PluginBag$LazyPluginHolder. So far its seems good, but I still get Exceptions when starting the Solr Cloud with the following errors:

SolrException: Blob loading failed: .no active replica available for .system collection...

In my case I'm using custom code for a couple of UpdateProcessors. So it seems, that this lazy mechanism is not working well for UpdateProcessors.

Inside the calzz LazyPluginHolder the comment says:

"A class that loads plugins Lazily. When the get() method is invoked the Plugin is initialized and returned."

When a core is initialized and you have a custom UpdateProcessor, the get-method is invoked directly and the lazy loading mechanism tries to get the custom class from the MemClassLoader, but in most scenarios the system collection is not up and the above Exception is thrown...

So maybe it’s the case that for UpdateProcessors while initializing a core, the routine is not implemented optimal for the lazy loading mechanism?

 

Here are the steps to reproduce the issue:

1. Unpack solr 7.3.0
   1.1 Add SOLR_OPTS="$SOLR_OPTS -Denable.runtime.lib=true" to solr.in.sh
   1.2 Start Solr with -c option
2. Setup .system collection:
   2.1 Upload custom test jar --> curl -X POST -H 'Content-Type: application/octet-stream' --data-binary @<path to custom jar>/update-processor-0.0.1-SNAPSHOT.jar http://<your solr>/solr/.system/blob/test_blob
   2.2 Alter maxShardsPerNode --> .../admin/collections?action=MODIFYCOLLECTION&collection=.system&maxShardsPerNode=2
   2.2 Add Replica to .system collection --> .../admin/collections?action=ADDREPLICA&collection=.system&shard=shard1
3. Setup test collection:
   3.1 Upload test conf to ZK --> ./zkcli.sh -zkhost <your ZK host> -cmd upconfig -confdir <your config dir> -confname test_conf
   3.2 Create a test1 collection with commented UP-chain inside solrconfig.xml via Admin UI
   3.3 Add blob to test collection --> curl http://<your Solr>/solr/test1/config -H 'Content-type:application/json' -d '{"add-runtimelib": { "name":"test_blob", "version":1 }}'
   3.4 Uncomment the UP-chain and upload test-conf again --> ./zkcli.sh -zkhost <your ZK host> -cmd upconfig -confdir <your config dir> -confname test_conf
   3.5 Reload test1 collection
   3.6 Everything should work as expected now (no erros are shown)
4. Restart SOLR
   4.1 Now you can see: SolrException: Blob loading failed: No active replica available for .system collection

Expected: The lazy loading mechanism should work for UpdateProcessors inside core init routine, but it isn't due to above Exception.

Sometimes you are lucky and the test1 collection will be initialize after the .system collection. But in ~90% of the time this isn't the case...

Let me know if you need further details here,

 

Johannes