[CONNECTORS-663] ManifoldCF needs the ability to not always check for deletion on a crawl

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: ManifoldCF 1.1.1
• Fix Version/s: ManifoldCF 1.2
• Component/s: Framework agents process
• Assignee: Karl Wright
• Reporter: Karl Wright
• Created: 11/Mar/13 11:51
• Updated: 16/Mar/13 21:18
• Resolved: 16/Mar/13 21:18

Description

The ManifoldCF framework's crawling model always brings the index in synch with the repository by the end of the job. Unfortunately, for many repositories, the incremental nature of ManifoldCF is lost in part because deletion tracking is not done by the repository. ManifoldCF could therefore benefit by the ability to have two different job run cycles: (1) A full run, as is done now, and (2) a partial run, which does not necessarily attempt to clean up deletions. This of course only makes sense if subsequent job runs have the ability to do the deletion cleanup.

In principle, I believe this can can work but has significant implications in the following areas:

• Job states - there needs to be a new set of job states corresponding to which type of job run is selected;
• UI - there needs to be a way of telling ManifoldCF what kind of job run is desired;
• API - same problem as UI;
• Job scheduling; we need the ability to determine what kind of job run is done when, which also has schema implications