[NUTCH-1746] OutOfMemoryError in Mappers

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.7
• Component/s: generator, injector

Environment

Nutch running in local mode with 4M+ domains in domain-urlfilter.txt

• Patch Info: Patch Available
• Assignee: Unassigned
• Reporter: Greg Padiasek
• Created: 03/Apr/14 13:56
• Updated: 19/Oct/17 21:21

Description

Initially I found that Generator was throwing OutOfMemoryError exception no matter how much RAM I allocated to JVM. I fixed the problem by moving URLFilters, URLNormalizers and ScoringFilters to top-level class as singletons and re-using them in all Generator mapper instances.

Then I found the same problem in Injector and applied analogical fix.

Now it seems that this issue may be common in all Nutch Mapper implementations.

I was wondering if it would it be possible to integrate this kind of change
in the upstream code base and potentially update all vulnerable Mapper classes.

Issue Links

• is related to: NUTCH-2407 Memory leak causing Nutch Server to run out of memory