[SOLR-12229] Harden exception handling in CdcrUpdateLogSynchronizer

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Component/s: CDCR
• Assignee: Varun Thacker
• Reporter: Varun Thacker
• Created: 16/Apr/18 17:19
• Updated: 28/Oct/20 20:37
• Resolved: 28/Oct/20 20:37

Description

In CdcrUpdateLogSynchronizer when we ask for the last processed version, if the call fails and we don't catch the exception the synchronizer can quit.

 

Here's an example from a Jenkins failure 

[junit4] 2> 2810643 WARN (cdcr-update-log-synchronizer-9646-thread-1) [ ] o.a.s.h.CdcrUpdateLogSynchronizer Caught unexpected exception
[junit4] 2> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://127.0.0.1:45384/solr/cdcr-source_shard1_replica_n1: SolrCore is loading
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244) ~[java/:?]
[junit4] 2> at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219) ~[java/:?]
[junit4] 2> at org.apache.solr.handler.CdcrUpdateLogSynchronizer$UpdateLogSynchronisation.run(CdcrUpdateLogSynchronizer.java:147) [java/:?]
[junit4] 2> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_162]
[junit4] 2> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_162]
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_162]
[junit4] 2> at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_162]
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_162]
[junit4] 2> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_162]
[junit4] 2> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]

We should audit the code usage and then harden the failure scenarios and deal with it more gracefully