Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-2075

Generate will not choose URL without distance marker

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Auto Closed
    • Affects Version/s: 2.3
    • Fix Version/s: 2.5
    • Component/s: generator
    • Environment:

      Using HBase as back-end Storage

    • Patch Info:
      Patch Available
    • Flags:
      Patch

      Description

      It appears that there is a bug about certain links where nutch erases all markers and not only the inject, generate, fetch, parse, update markers but also the distance marker.

      The problem is that Nutch Generator doesn't check the validity of the marker distance (check if it's null) and keep wrong links (without the distance marker) in the GeneratorMapper. When the distance filter is activated, GeneratorMapper choose also URL without markers and so it doesn't repect the limit.

      I think it's in relation with the problem mention here : NUTCH-1930.

      This patch doesn't solved the problem which is all markers are erased (without any reasons apparently ..). But it can allow to stop the crawl...

      In order to find a solution about stopping crawl with problematics URL, I proposed this solution which is simply to avoid the URL when the distance marker is NULL.

      (Sorry if i put the code here)

      crawl/GeneratorMapper.java (initial code)
      // filter on distance
          if (maxDistance > -1) {
            CharSequence distanceUtf8 = page.getMarkers().get(DbUpdaterJob.DISTANCE);
            if (distanceUtf8 != null) {
              int distance = Integer.parseInt(distanceUtf8.toString());
              if (distance > maxDistance) {
                return;
              }
            }
          }
      
      
      crawl/GeneratorMapper.java (patch code)
      // filter on distance
          if (maxDistance > -1) {
            CharSequence distanceUtf8 = page.getMarkers().get(DbUpdaterJob.DISTANCE);
            if (distanceUtf8 != null) {
              int distance = Integer.parseInt(distanceUtf8.toString());
              if (distance > maxDistance) {
                return;
              }
            }
            else
            {
              // No distance marker, URL problem
              return;
            }
          }
      
      

      Example of link where the problem appears (put an http.content.limit highter than the content-length PDF) :
      http://www.annales.org/archives/x/marchal2.pdf

      Hope it can help ...

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Catharcis Alexandre Demeyer
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: