[SCB-1450] Microservice instance may get stuck when watch connection to sc is recovered and pulling instance action is executed in eventloop thread

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: java-chassis-1.3.0
• Component/s: Java-Chassis
• Assignee: Unassigned
• Reporter: Liu HuaiZhou
• Created: 20/Aug/19 02:38
• Updated: 16/Sep/19 01:50
• Resolved: 29/Aug/19 12:39

Description

If you set property of servicecomb.service.registry.instance.watch to true and then set the peoperty of servicecomb.service.registry.client.eventLoopPoolSize to the smaller value as you can ,like 2. Unluckily, service center crashed..The following poor scenario  you may fall in:

1 You had been informed while Service Center recovered a  few minutes later.It is so sad ,you can't get micro-service instance information from the Service Center.

2 You can never  to shutdown your micro-service instance gracefully,

Following are the dump logs:

[ WARN ] [2019-06-11 22:32:38.417] [178.42.0.21] [vertx-blocked-thread-checker] [io.vertx.core.impl.BlockedThreadChecker$1.run(BlockedThreadChecker.java:55)] - Thread Thread[registry-vert.x-eventloop-thread-2,5,main] has been blocked for 20417526 ms, time limit is 2000 ms
io.vertx.core.VertxException: Thread blocked
at sun.misc.Unsafe.park(Native Method) ~[?:1.8.0_212]
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) ~[?:1.8.0_212]
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) ~[?:1.8.0_212]
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) ~[?:1.8.0_212]
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304) ~[?:1.8.0_212]
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231) ~[?:1.8.0_212]
at org.apache.servicecomb.serviceregistry.client.http.ServiceRegistryClientImpl.findServiceInstances(ServiceRegistryClientImpl.java:739) ~[service-registry-1.0.0.B003.H69.jar:1.0.0.B003.H69]
at org.apache.servicecomb.serviceregistry.registry.AbstractServiceRegistry.findServiceInstances(AbstractServiceRegistry.java:236) ~[service-registry-1.0.0.B003.H69.jar:1.0.0.B003.H69]
at org.apache.servicecomb.serviceregistry.RegistryUtils.findServiceInstances(RegistryUtils.java:231) ~[service-registry-1.0.0.B003.H69.jar:1.0.0.B003.H69]
at org.apache.servicecomb.serviceregistry.consumer.MicroserviceVersions.pullInstances(MicroserviceVersions.java:138) ~[service-registry-1.0.0.B003.H69.jar:1.0.0.B003.H69]
at org.apache.servicecomb.serviceregistry.consumer.MicroserviceVersions.submitPull(MicroserviceVersions.java:130) ~[service-registry-1.0.0.B003.H69.jar:1.0.0.B003.H69]
at org.apache.servicecomb.serviceregistry.consumer.MicroserviceManager.refreshInstances(MicroserviceManager.java:99) ~[service-registry-1.0.0.B003.H69.jar:1.0.0.B003.H69

Issue Links

• is related to: SCB-1461 servicecomb.service.registry.client.workerPoolSize hasn't display the real role that it has announce by its name.
• is related to: SCB-1462 Two properties has been required by fix SCB-1450.
• links to: GitHub Pull Request #1301