Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-2229

Allow Jexl expressions on CrawlDatum's fixed attributes

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.11
    • Fix Version/s: 1.12
    • Component/s: crawldb
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      CrawlDatum allows Jexl expressions on its metadata fields nicely, but it lacks the opportunity to select on attributes like fetchTime and modifiedTime.

      This includes a rudimentary date parser only supporting the yyyy-MM-dd'T'HH:mm:ss'Z' format:

      Dump everything with a modifiedTime higher than 2016-03-20T00:00:00Z

      bin/nutch readdb crawl/crawldb/ -dump out -format csv -expr "(modifiedTime > 2016-03-20T00:00:00Z)"
      

      Dump everything that is an HTML file

      bin/nutch readdb crawl/crawldb/ -dump out -format csv -expr "(Content_Type == 'text/html' || Content_Type == 'application/xhtml+xml')"
      

      Keep in mind:

      • Jexl doesn't allow a hyphen/minus in field identifier, they are transformed to underscores
      • string literals must be in quotes, only surrounding qoute needs to be escaped by backslash

        Attachments

        1. NUTCH-2229.patch
          4 kB
          Markus Jelsma

          Activity

            People

            • Assignee:
              markus17 Markus Jelsma
              Reporter:
              markus17 Markus Jelsma
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: