Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-201

add support for subcollections

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 0.8
    • 0.8
    • None
    • None

    Description

      Subcollection is a subset of an index. Subcollections are defined
      by urlpatterns in form of white/blacklist. So to get the page into
      subcollection it must match the whitelist and not the blacklist.

      Subcollection definitions are read from a file subcollections.xml
      and the format is as follows (imagine here that you are crawling all
      the virtualhosts from apache.org and you wan't to tag pages with
      url pattern "http://lucene.apache.org/" to be part of subcollection
      lucene.

      <?xml version="1.0" encoding="UTF-8"?>
      <subcollections>
      <subcollection>
      <name>lucene</name>
      <id>lucene</id>
      <whitelist>http://lucene.apache.org/</whitelist>
      <blacklist />
      </subcollection>
      </subcollections>

      plugin contains indexingfilter, query filter and supporting classes

      Attachments

        1. subcollections-1.patch
          21 kB
          Sami Siren
        2. subcollections.2.patch
          25 kB
          Sami Siren

        Activity

          People

            siren Sami Siren
            siren Sami Siren
            Votes:
            4 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: