Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-2227

RegexParseFilter

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.11
    • Fix Version/s: 1.12
    • Component/s: parser
    • Labels:
      None

      Description

      A parse filter that takes a regex and a field name. If regex matches via matcher.find() on the HTML. The field name is set to true in the CrawlDatum's metadata.

      Combined with the HostDB, it is easy to get a list of hosts that match some regex criteria.

      # Example configuration file for parsefilter-regex
      #
      # Parse metadata field <name> is set to true if the HTML matches the regex. The
      # source can either be html or text. If source is html, the regex is applied to
      # the entire HTML tree. If source is text, the regex is applied to the
      # extracted text.
      #
      # format: <name>\t<source>\t<regex>\n
      

        Attachments

        1. NUTCH-2227.patch
          21 kB
          Markus Jelsma
        2. NUTCH-2227.patch
          20 kB
          Markus Jelsma
        3. NUTCH-2227.patch
          19 kB
          Markus Jelsma
        4. NUTCH-2227.patch
          18 kB
          Markus Jelsma
        5. NUTCH-2227.patch
          17 kB
          Markus Jelsma

          Activity

            People

            • Assignee:
              markus17 Markus Jelsma
              Reporter:
              markus17 Markus Jelsma
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: