[SOLR-14706] Upgrading 8.6.0 to 8.6.1 causes collection creation to fail

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 8.6.1, 8.7
• Fix Version/s: 8.6.1, 8.7
• Component/s: AutoScaling

Environment

8.6.1 upgraded from 8.6.0 with more than one node

• Assignee: Houston Putman
• Reporter: Gus Heck
• Created: 03/Aug/20 12:45
• Updated: 19/Aug/20 21:25
• Resolved: 13/Aug/20 14:52

Description

The following steps will reproduce a situation in which collection creation fails with this stack trace:

2020-08-03 12:17:58.617 INFO  (OverseerThreadFactory-22-thread-1-processing-n:192.168.2.106:8981_solr) [   ] o.a.s.c.a.c.CreateCollectionCmd Create collection test861
2020-08-03 12:17:58.751 ERROR (OverseerThreadFactory-22-thread-1-processing-n:192.168.2.106:8981_solr) [   ] o.a.s.c.a.c.OverseerCollectionMessageHandler Collection: test861 operation: create failed:org.apache.solr.common.SolrException
	at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:347)
	at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:264)
	at org.apache.solr.cloud.OverseerTaskProcessor$Runner.run(OverseerTaskProcessor.java:517)
	at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:212)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Only one extra tag supported for the tag cores in {
  "cores":"#EQUAL",
  "node":"#ANY",
  "strict":"false"}
	at org.apache.solr.client.solrj.cloud.autoscaling.Clause.<init>(Clause.java:122)
	at org.apache.solr.client.solrj.cloud.autoscaling.Clause.create(Clause.java:235)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
	at org.apache.solr.client.solrj.cloud.autoscaling.Policy.<init>(Policy.java:144)
	at org.apache.solr.client.solrj.cloud.autoscaling.AutoScalingConfig.getPolicy(AutoScalingConfig.java:372)
	at org.apache.solr.cloud.api.collections.Assign.usePolicyFramework(Assign.java:300)
	at org.apache.solr.cloud.api.collections.Assign.usePolicyFramework(Assign.java:277)
	at org.apache.solr.cloud.api.collections.Assign$AssignStrategyFactory.create(Assign.java:661)
	at org.apache.solr.cloud.api.collections.CreateCollectionCmd.buildReplicaPositions(CreateCollectionCmd.java:415)
	at org.apache.solr.cloud.api.collections.CreateCollectionCmd.call(CreateCollectionCmd.java:192)
	... 6 more

Generalized steps:

1. Deploy 8.6.0 with separate data directories, create a collection to prove it's working
2. download https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-8.6.1-RC1-reva32a3ac4e43f629df71e5ae30a3330be94b095f2/solr/solr-8.6.1.tgz
3. Stop the server on all nodes
4. replace the 8.6.0 with 8.6.1
5. Start the server
6. via the admin UI create a collection
7. Observe failure warning box (with no text), check logs, find above trace

Or more exactly here are my actual commands with a checkout of the 8.6.0 tag in the working dir to which cloud.sh was configured:

1. /cloud.sh new -r upgrademe
2. Create collection named test860 via admin ui with _default
3. ./cloud.sh stop
4. cd upgrademe/
5. cp ../8_6_1_RC1/solr-8.6.1.tgz .
6. mv solr-8.6.0-SNAPSHOT old
7. tar xzvf solr-8.6.1.tgz
8. cd ..
9. ./cloud.sh start
10. Try to create collection test861 with _default config

For those not familiar with it the first command there with cloud.sh builds the tarball in the working directory and then makes a directory named "upgrademe" copies it to "upgrademe" unpacks it, sets up a chroot based on the path in (already running separate) zookeeper, and by default starts 4 local nodes on ports 8981 to 8984 all with separate data directorys hosted under the "upgrademe" directory.

Issue Links

• links to: GitHub Pull Request #1716
• links to: GitHub Pull Request #1739