[CONNECTORS-1122] Explore ways to make job start be faster in systems with lots of documents

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: ManifoldCF 1.8, ManifoldCF 2.0
• Fix Version/s: ManifoldCF 1.9, ManifoldCF 2.1
• Component/s: Framework crawler agent
• Assignee: Karl Wright
• Reporter: Karl Wright
• Created: 15/Dec/14 12:21
• Updated: 13/Jan/15 12:44
• Resolved: 13/Jan/15 12:44

Description

Job start requires all documents to be marked as needing reprioritization now. We should consider ways in which we can reduce the need to do this as much as possible. For example, if there are NO documents at all for a job, reprioritization is by definition unneeded. Alternatively, coming up with a way of determining if there are any bin-level overlaps between documents made active by a job start at documents elsewhere, we could be more targeted.