CrawlDatum needs more status codes, e.g. to reflect redirected pages. However, current values of status codes are linear, which prevents us from adding new codes in proper places. This is also related to the logic in CrawlDbReducer, which makes decisions based on arithmetic ordering of status code values.
I propose to change the codes so that they are grouped into related values, with significant gaps between groups for adding new codes without causing significant reordering. I also propose to change the logic in CrawlDbReducer so that its operation is not so dependent on actual code values.
A mapping should also be added between old and new codes to facilitate backward-compatibility of existing data. This mapping should be applied on the fly, without requiring explicit data conversion.
Improvement
Closed
Major
CrawlDatum status and CrawlDbReducer refactoring