Nutch
  1. Nutch
  2. NUTCH-802

Problems managing outlinks with large url length

    Details

    • Patch Info:
      Patch Available

      Description

      Nutch can get idle during the collection of outlinks if the URL address of the outlink is too large.

      The maximum sizes of an URL for the main web servers are:

      • Apache: 4,000 bytes
      • Microsoft Internet Information Server (IIS): 16, 384 bytes
      • Perl HTTP::Daemon: 8.000 bytes

      URL adress sizes bigger than 4000 bytes are problematic, so the limit should be set in the nutch-default.xml configuration file.

      I attached a patch

        Activity

          People

          • Assignee:
            Andrzej Bialecki
            Reporter:
            Pablo Aragón
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development