[CONNECTORS-1746] Adding conditions to execute PostgreSQL's ANALYZE command to avoid crawling become extremely slow.

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: ManifoldCF 2.25
• Component/s: Web connector

Environment

Using ManifoldCF 2.24 with PostgreSQL 12.14 as the database. 

• Assignee: Karl Wright
• Reporter: Mingchun Zhao
• Created: 07/May/23 01:06
• Updated: 01/Jun/23 16:05
• Resolved: 01/Jun/23 16:05

Description

Sometimes, the crawling does not process any documents for a while and there is nothing logged about long-running queries. The performance can be restored by firing the 'ANALYZE' command manually. It seems that a bad query plan caused this performance problem.

Therefore, in addition to the current configuration parameter 'org.apache.manifoldcf.db.postgres.analyze.<tablename>', it is considered necessary to execute the 'ANALYZE' even in the following situations.
1. When the number of records in the table exceeds the number required for creating a execution plan after the job starts.
2. When the crawling performance slows down. For example, if the processing rate of documents drops below a specified threshold.