Nutch
  1. Nutch
  2. NUTCH-1142

Normalization and filtering in WebGraph

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      The WebGraph programs performs URL normalization. Since normalization of outlinks is already performed during the parse it should become optional. There is also no URL filtering mechanism in the web graph program. When a CrawlDatum is removed from the CrawlDB by an URL filter is should be possible to remove it from the web graph as well.

      1. NUTCH-1142-1.5-3.patch
        8 kB
        Markus Jelsma
      2. NUTCH-1142-1.5-2.patch
        7 kB
        Markus Jelsma
      3. NUTCH-1142-1.4.patch
        6 kB
        Markus Jelsma

        Issue Links

          Activity

          Markus Jelsma created issue -
          Markus Jelsma made changes -
          Field Original Value New Value
          Link This issue is duplicated by NUTCH-1144 [ NUTCH-1144 ]
          Markus Jelsma made changes -
          Summary Normalization optional in WebGraph Normalization and filtering in WebGraph
          Patch Info Patch Available [ 10042 ]
          Description The WebGraph programs performs URL normalization. Since normalization of outlinks is already performed during the parse it should become optional. The WebGraph programs performs URL normalization. Since normalization of outlinks is already performed during the parse it should become optional. There is also no URL filtering mechanism in the web graph program. When a CrawlDatum is removed from the CrawlDB by an URL filter is should be possible to remove it from the web graph as well.
          Markus Jelsma made changes -
          Attachment NUTCH-1142-1.4.patch [ 12497450 ]
          Markus Jelsma made changes -
          Attachment NUTCH-1142-1.5-2.patch [ 12498147 ]
          Markus Jelsma made changes -
          Link This issue is duplicated by NUTCH-1171 [ NUTCH-1171 ]
          Markus Jelsma made changes -
          Attachment NUTCH-1142-1.5-3.patch [ 12498762 ]
          Markus Jelsma made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Lewis John McGibbney made changes -
          Status Resolved [ 5 ] Closed [ 6 ]

            People

            • Assignee:
              Markus Jelsma
              Reporter:
              Markus Jelsma
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development