[CONNECTORS-781] Fault-Tolerant Setup for ManifoldCF Agent.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: ManifoldCF 1.5
• Fix Version/s: ManifoldCF 1.5
• Component/s: Framework agents process, Framework core, Framework crawler agent
• Labels: agents crawler fault-tolerance
• Assignee: Karl Wright
• Reporter: Swami Rajamohan
• Created: 24/Sep/13 19:30
• Updated: 19/Dec/13 12:46
• Resolved: 08/Dec/13 19:02

Description

It should be possible to setup ManifoldCF as a Fault-Tolerant infrastructure.
The Agent component of ManifoldCF should support multiple instances of an agent crawling against a single crawl store, to be able to both distribute (share) the crawl load as well as to be able to pick up a request that gets abruptly terminated due to either partitioning of the instance/failure of the instance itself.
Since there is a proposal to move to a store like Voldemort, it would be nice to be able to have a fault tolerant infrastructure.

Issue Links

• contains: CONNECTORS-180 Connector factories all have a Pool class that should be derived from a base Pool class
• depends upon: CONNECTORS-13 We should augment file-based synchronization to allow a process/service-based synchronization as well
• is duplicated by: CONNECTORS-780 For scalability/reliability, allow multiple agents processes to run at the same time
• relates to: CONNECTORS-830 Framework-based throttling is not cross-cluster
• relates to: CONNECTORS-829 Add cross-cluster throttling support for Web and RSS connectors