Minor Description
If we run more than one yardstick driver, benchmark hangs intermittently.
We've got yardstick's base driver class org.apache.ignite.yardstick.IgniteAbstractBenchmark it has logic to wait all the nodes in the cluster.
final CountDownLatch nodesStartedLatch = new CountDownLatch(1);
ignite().events().localListen(new IgnitePredicate<Event>() {
@Override public boolean apply(Event gridEvt) {
if (nodesStarted())
nodesStartedLatch.countDown();
return true;
}
}, EVT_NODE_JOINED);
if (!nodesStarted()) {
println(cfg, "Waiting for " + (args.nodes() - 1) + " nodes to start...");
nodesStartedLatch.await();
}
This code is executed on every driver node.
If we want to close local ignite instance just after cluster is ready (containing expected number of nodes), sometimes we'll have dead lock:
1) cluster contains N-1 nodes, all nodes are waiting for the Nth node.
2) Nth node is connected, cluster receives message, waitForNodes code of Nth node is not executed.
3) N-1 nodes got this message and stop waiting.
4) N-1 thinks that cluster is ready and call ignite.close() on their local instances
5) Nth node starts waiting for cluster to contain number of nodes, but N-1 of them closed their instances
6) Nth node is waiting infinitely.
We can avoid this problem if we use distributed CountDownLatch
Attachments
Issue Links
- links to
