[CONNECTORS-1599] response code 401 still gets deleted with the setting "keep unreachable documents"

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: ManifoldCF 2.12
• Fix Version/s: ManifoldCF 2.13
• Component/s: Web connector
• Assignee: Karl Wright
• Reporter: roel goovaerts
• Created: 11/Apr/19 09:44
• Updated: 18/Apr/19 09:11
• Resolved: 11/Apr/19 12:46

Description

Even with the "Hop count mode" set to "keep unreachable documents, 'for now' || forever" manifold deletes documents for which it receives a 401 response code.

The documentation does not specify such a distinction as described above. Is there some information/configuration that I'm missing? Is there a reasoning behind the guaranteed deletion of a 401?

Ideally, for our use-case, we would want to remove all documents that return 404, but keep everything which is due the server not responding or the crawler being unauthenticated.

Is there a way to configure this in a more granular fashion?

Regards,
roel