Uploaded image for project: 'Nutch'
  1. Nutch
  2. NUTCH-1115

Option to disable fixing of embedded params in DomContentUtils

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.3
    • 1.4
    • parser
    • None

    Description

      Add option to disable fixing of embedded params:
      http://lucene.472066.n3.nabble.com/Outlinks-with-embedded-params-td3332396.html

      When enabled, millions of crap url's are output as outlink. This results in many 404 in the DB and many very long URL's that actually lead to the same page.

      Attachments

        1. NUTCH-1115-1.4-2.patch
          3 kB
          Markus Jelsma
        2. NUTCH-1115-1.4-1.patch
          1 kB
          Markus Jelsma

        Issue Links

          is part of

          Bug - A problem which impairs or prevents the functions of the product. NUTCH-797 URL not properly constructed when link target begins with a "?"

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. NUTCH-436 Incorrect handling of relative paths when the embedded URL path is empty

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed

          Activity

            People

              markus17 Markus Jelsma
              markus17 Markus Jelsma
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: