Nutch
  1. Nutch
  2. NUTCH-1314

Impose a limit on the length of outlink target urls

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 2.4
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      In the past we have encountered situations where crawling specific broken sites resulted in ridiciously long urls that caused the stalling of tasks. The regex plugins (normalizing/filtering) processed single urls for hours, if not indefinitely hanging.

      My suggestion is to limit the outlink url target length as soon possible. It is a configurable limit, the default is 3000. This should be reasonably long enough for most uses. But sufficienly strict enough to make sure regex plugins do not choke on urls that are too long. Please see attached patch for the Nutchgora implementation.

      I'd like to hear what you think about this.

      1. NUTCH-1314.patch
        2 kB
        Ferdy Galema
      2. NUTCH-1314-trunk.patch
        3 kB
        Lewis John McGibbney
      3. NUTCH-1314-v2.patch
        4 kB
        Lewis John McGibbney
      4. NUTCH-1314-v3.patch
        2 kB
        Canan Girgin

        Issue Links

          Activity

          Ferdy Galema created issue -
          Ferdy Galema made changes -
          Field Original Value New Value
          Attachment NUTCH-1314.patch [ 12518699 ]
          Julien Nioche made changes -
          Fix Version/s 1.7 [ 12323281 ]
          Fix Version/s 2.2 [ 12323285 ]
          Fırat KÜÇÜK made changes -
          Link This issue is related to NUTCH-1106 [ NUTCH-1106 ]
          Fırat KÜÇÜK made changes -
          Link This issue is related to NUTCH-1531 [ NUTCH-1531 ]
          Lewis John McGibbney made changes -
          Attachment NUTCH-1314-v2.patch [ 12580639 ]
          Attachment NUTCH-1314-trunk.patch [ 12580640 ]
          Lewis John McGibbney made changes -
          Fix Version/s 2.3 [ 12324325 ]
          Fix Version/s 1.7 [ 12323281 ]
          Fix Version/s 2.2 [ 12323285 ]
          Canan Girgin made changes -
          Attachment NUTCH-1314-v3.patch [ 12589916 ]
          Lewis John McGibbney made changes -
          Fix Version/s 2.4 [ 12324540 ]
          Fix Version/s 2.3 [ 12324325 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Ferdy Galema
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Development