Details
-
Bug
-
Status: Reopened
-
Critical
-
Resolution: Unresolved
-
None
-
None
-
None
Description
When we update the index,can not guarantee that the contents which be indexed is the latest.In the class IndexerMapReduce and method reduce(), it has the following code:
public void reduce(Text key, Iterator<NutchWritable> values,
OutputCollector<Text, NutchDocument> output, Reporter reporter) throws IOException
else if (value instanceof ParseData)
{ parseData = (ParseData)value; }else if (value instanceof ParseText)
{ parseText = (ParseText)value; } ……
}
For example,30 days ago,I fetched the web page A,now I fetch it again. Then the key A will correspond to two ParseData objects(located in different segments).But in this code,it does not compare the fetch time and simply overwrites the previous value.So the final value maybe the old one.
Attachments
Issue Links
- relates to
-
NUTCH-1617 IndexerMapReduce to consider latest fetchDatum
- Open
-
NUTCH-1625 IndexerMapReduce skips FETCH_NOTMODIFIED
- Closed