Affects Version/s: None
Fix Version/s: nutchgora
Patch Info:Patch Available
GORA-120 (this already is a huge performance boost by itself) one of the major bottlenecks of the DbUpdaterReducer is the deletion of the markers. The update reducer simply sets every row to delete its markers. A lot of rows do not actually have the markers but the deletes are fired away in any case. Because the markers are already always on the input, a simple check to see if they exist greaty improves performance.
In particular it is very expensive in HBase, because every single Delete inmediately triggers a connection to the regionservers. (They ignore the "autoflush=false" directive). Although deletes can be done in batch, this is currently not supported by Gora. For one it is very difficult to implement in the current HBaseStore with regard to multithreading, and secondly I noticed performance did not increase significantly.
By performance debugging on a real life cluster this currently seems to be the biggest bottleneck of the DbUpdaterReducer. (Remember only after applying
|Status||Open [ 1 ]||Closed [ 6 ]|
|Resolution||Fixed [ 1 ]|
|Transition||Time In Source Status||Execution Times||Last Executer||Last Execution Date|
|7d 18h 24m||1||Ferdy Galema||26/Apr/12 09:02|