[NUTCH-51] Removing a plugin after fetch but before indexing causes errors

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Component/s: indexer

Environment

Linux/Jdk 1.4.2

• Assignee: Unassigned
• Reporter: byron miller
• Created: 25/Apr/05 07:07
• Updated: 25/May/06 17:27
• Resolved: 24/May/05 03:45

Description

I created a segment of 4 million pages with a normal fetch. I created an index on this segment and it took over 30 hours to complete - when it did i had errors running a search that the nutchbean couldn't find the title value.

I did some researching and it looks like i should have ran a dedup or something else.

So to debug i tried to re-index without some of the plugins enabled, and that is when i get this error:

root@elect [/home2/mozdex/nutch]# bin/nutch segread -fix segments/20050420222001/
run with heapsize 1500
-Xmx1500m
050424 180125 parsing file:/home2/mozdex/nutch/conf/nutch-default.xml
050424 180126 parsing file:/home2/mozdex/nutch/conf/nutch-site.xml
050424 180126 No FS indicated, using default:local
root@elect [/home2/mozdex/nutch]# bin/nutch index segments/20050420222001 -dir /home/work
run with heapsize 1500
-Xmx1500m
050424 180152 parsing file:/home2/mozdex/nutch/conf/nutch-default.xml
050424 180152 parsing file:/home2/mozdex/nutch/conf/nutch-site.xml
050424 180152 No FS indicated, using default:local
050424 180152 indexing segment: segments/20050420222001
050424 180152 * Opening segment 20050420222001
050424 180153 * Indexing segment 20050420222001
050424 180153 Plugins: looking in: /home2/mozdex/nutch/build/plugins
050424 180154 parsing: /home2/mozdex/nutch/build/plugins/protocol-http/plugin.xml
050424 180154 parsing: /home2/mozdex/nutch/build/plugins/parse-html/plugin.xml
050424 180154 parsing: /home2/mozdex/nutch/build/plugins/parse-text/plugin.xml
050424 180154 parsing: /home2/mozdex/nutch/build/plugins/parse-pdf/plugin.xml
050424 180154 not including: /home2/mozdex/nutch/build/plugins/parse-msword
050424 180154 not including: /home2/mozdex/nutch/build/plugins/parse-ext
050424 180154 not including: /home2/mozdex/nutch/build/plugins/index-basic
050424 180154 parsing: /home2/mozdex/nutch/build/plugins/query-basic/plugin.xml
050424 180154 not including: /home2/mozdex/nutch/build/plugins/query-more
050424 180154 parsing: /home2/mozdex/nutch/build/plugins/query-site/plugin.xml
050424 180154 parsing: /home2/mozdex/nutch/build/plugins/query-url/plugin.xml
050424 180154 parsing: /home2/mozdex/nutch/build/plugins/urlfilter-regex/plugin.xml
050424 180154 not including: /home2/mozdex/nutch/build/plugins/urlfilter-prefix
050424 180154 not including: /home2/mozdex/nutch/build/plugins/creativecommons
050424 180154 not including: /home2/mozdex/nutch/build/plugins/language-identifier
050424 180154 parsing: /home2/mozdex/nutch/build/plugins/clustering-carrot2/plugin.xml
050424 180155 parsing: /home2/mozdex/nutch/build/plugins/ontology/plugin.xml
Exception in thread "main" java.lang.ExceptionInInitializerError
at org.apache.nutch.indexer.IndexSegment.indexPages(IndexSegment.java:145)
at org.apache.nutch.indexer.IndexSegment.main(IndexSegment.java:254)
Caused by: java.lang.RuntimeException: org.apache.nutch.indexer.IndexingFilter not found.
at org.apache.nutch.indexer.IndexingFilters.<clinit>(IndexingFilters.java:36)
... 2 more

It seems when you generate a fetchlist with the plugin you can't process the index without the plugin.

If this is true, a friendly error message would be nice and perhaps a utility to "fix" the segment so it can be processed without the plugin.

Issue Links

• is related to: NUTCH-284 NullPointerException during index