Details
-
New Feature
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
Description
We'd like to understand how applications built on top of ZooKeeper behave under various faulty conditions, which is important to build resilient end to end solutions and avoid ZooKeeper being single point of failure. We'd also like to achieve this in both unit tests (in process) and integration tests (in and out of process). Traditional methods of using external fault injection mechanisms are non deterministic and requires non trivial set up and hard to integrate with unit tests, so here we introduce the ZooKeeperController service which solves both.
The basic idea here is to create a controllable ZooKeeperServer which accepts various control commands (such as - delay request, drop request, eat request, expire session, shutdown, trigger leader election, and so on), and reacting based on incoming commands. The controllable server and production server share the same underlying machineries (quorum peers, ZooKeeper server, etc) but code paths are separate, thus this feature has no production impact.
This controller system is currently composed of following pieces:
- CommandClient: a convenient HTTP client to send control commands to controller service.
- CommandListener: an embed HTTP server listening incoming commands and dispatch to controller service.
- Controller Service: the service that's responsible to create controllable ZK server and the controller.
- ZooKeeperServerController: the controller that changes the behavior of ZK server runtime.
- Controllable Cnx / Factory: controllable connection that accept behavior change requests.
In future more control commands and controllable components can be added on top of this framework.
This can be used in either unit tests / integration tests as an in process embedded controllable ZooKeeper server, or as an out of process stand alone controllable ZooKeeper process.
Attachments
Issue Links
- links to
