Minor Description
To generate a document signature (MD5 digest) of the binary content requires that documents are parsed during the parse or fetch step. The signature is required for deduplication and detection of unmodified content and should be always available, also in a workflow which does not require that documents are parsed, e.g., because HTML content is exported to WARC files.
Attachments
Issue Links
- links to