Major Description
It would be nice to have a Pub/Sub model in Nutch to emit certain events (ex- Fetcher events like fetch-start, fetch-end, a fetch report which may contain data like outlinks of the current fetched url, score, etc).
A consumer of this functionality could use this data to generate real time visualization and generate statics of the crawl without having to wait for the fetch round to finish.
The REST API could contain an endpoint which would respond with a url to which a client could subscribe to get the fetcher events.
Attachments
Attachments
Issue Links
- links to
