Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6
    • Component/s: None
    • Labels:
      None
    • Patch Info:
      Patch Available

      Description

      Nutch would benefit from having a host normalizer. A host normalizer maps a given host to the desired host. A basic example is to map www.apache.org to apache.org. The Apache website is one of many on the internet that has a duplicate website on the same domain just because it allows both www and non-www to return HTTP 200 and proper content.

      It is also able to handle wildcards such as *.example.org to example.org if there are multiple sub domains that actually point to the same website.

      Large internet crawls tend to get polluted very quickly due to these problems. It also leads to skewed scores in the webgraph as different websites link to different versions of the same duplicate website. An example:

      # Force all sub domains to non-www.
      *.example.com example.com
      
      # Force www sub domain to non-www.
      www.example.net example.net
      
      # Force non-www. sub domain to www
      example.org www.example.org
      

        Issue Links

          Activity

          Markus Jelsma created issue -
          Markus Jelsma made changes -
          Field Original Value New Value
          Patch Info Patch Available [ 10042 ]
          Hide
          Markus Jelsma added a comment -

          Patch for 1.5.

          Show
          Markus Jelsma added a comment - Patch for 1.5.
          Markus Jelsma made changes -
          Attachment NUTCH-1319-1.5-1.patch [ 12519415 ]
          Hide
          Markus Jelsma added a comment -

          Objections? I'd like to resolve this issue.

          Show
          Markus Jelsma added a comment - Objections? I'd like to resolve this issue.
          Markus Jelsma made changes -
          Link This issue blocks NUTCH-1326 [ NUTCH-1326 ]
          Hide
          Markus Jelsma added a comment -

          20120304-push-1.6

          Show
          Markus Jelsma added a comment - 20120304-push-1.6
          Markus Jelsma made changes -
          Fix Version/s 1.6 [ 12319941 ]
          Fix Version/s 1.5 [ 12318246 ]
          Hide
          Markus Jelsma added a comment -

          Committed for 1.6 in rev. 1349236.

          Show
          Markus Jelsma added a comment - Committed for 1.6 in rev. 1349236.
          Markus Jelsma made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in nutch-trunk-maven #310 (See https://builds.apache.org/job/nutch-trunk-maven/310/)
          NUTCH-1319 HostNormalizer plugin (Revision 1349236)

          Result = SUCCESS
          markus :
          Files :

          • /nutch/trunk/CHANGES.txt
          • /nutch/trunk/conf/host-urlnormalizer.txt
          • /nutch/trunk/src/plugin/urlnormalizer-host
          • /nutch/trunk/src/plugin/urlnormalizer-host/build.xml
          • /nutch/trunk/src/plugin/urlnormalizer-host/data
          • /nutch/trunk/src/plugin/urlnormalizer-host/data/hosts.txt
          • /nutch/trunk/src/plugin/urlnormalizer-host/ivy.xml
          • /nutch/trunk/src/plugin/urlnormalizer-host/plugin.xml
          • /nutch/trunk/src/plugin/urlnormalizer-host/src
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/java
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer/host
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer/host/HostURLNormalizer.java
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/test
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer/host
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer/host/TestHostURLNormalizer.java
          Show
          Hudson added a comment - Integrated in nutch-trunk-maven #310 (See https://builds.apache.org/job/nutch-trunk-maven/310/ ) NUTCH-1319 HostNormalizer plugin (Revision 1349236) Result = SUCCESS markus : Files : /nutch/trunk/CHANGES.txt /nutch/trunk/conf/host-urlnormalizer.txt /nutch/trunk/src/plugin/urlnormalizer-host /nutch/trunk/src/plugin/urlnormalizer-host/build.xml /nutch/trunk/src/plugin/urlnormalizer-host/data /nutch/trunk/src/plugin/urlnormalizer-host/data/hosts.txt /nutch/trunk/src/plugin/urlnormalizer-host/ivy.xml /nutch/trunk/src/plugin/urlnormalizer-host/plugin.xml /nutch/trunk/src/plugin/urlnormalizer-host/src /nutch/trunk/src/plugin/urlnormalizer-host/src/java /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer/host /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer/host/HostURLNormalizer.java /nutch/trunk/src/plugin/urlnormalizer-host/src/test /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer/host /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer/host/TestHostURLNormalizer.java
          Hide
          Hudson added a comment -

          Integrated in Nutch-trunk #1869 (See https://builds.apache.org/job/Nutch-trunk/1869/)
          NUTCH-1319 HostNormalizer plugin (Revision 1349236)

          Result = SUCCESS
          markus : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349236
          Files :

          • /nutch/trunk/CHANGES.txt
          • /nutch/trunk/conf/host-urlnormalizer.txt
          • /nutch/trunk/src/plugin/urlnormalizer-host
          • /nutch/trunk/src/plugin/urlnormalizer-host/build.xml
          • /nutch/trunk/src/plugin/urlnormalizer-host/data
          • /nutch/trunk/src/plugin/urlnormalizer-host/data/hosts.txt
          • /nutch/trunk/src/plugin/urlnormalizer-host/ivy.xml
          • /nutch/trunk/src/plugin/urlnormalizer-host/plugin.xml
          • /nutch/trunk/src/plugin/urlnormalizer-host/src
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/java
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer/host
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer/host/HostURLNormalizer.java
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/test
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer/host
          • /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer/host/TestHostURLNormalizer.java
          Show
          Hudson added a comment - Integrated in Nutch-trunk #1869 (See https://builds.apache.org/job/Nutch-trunk/1869/ ) NUTCH-1319 HostNormalizer plugin (Revision 1349236) Result = SUCCESS markus : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1349236 Files : /nutch/trunk/CHANGES.txt /nutch/trunk/conf/host-urlnormalizer.txt /nutch/trunk/src/plugin/urlnormalizer-host /nutch/trunk/src/plugin/urlnormalizer-host/build.xml /nutch/trunk/src/plugin/urlnormalizer-host/data /nutch/trunk/src/plugin/urlnormalizer-host/data/hosts.txt /nutch/trunk/src/plugin/urlnormalizer-host/ivy.xml /nutch/trunk/src/plugin/urlnormalizer-host/plugin.xml /nutch/trunk/src/plugin/urlnormalizer-host/src /nutch/trunk/src/plugin/urlnormalizer-host/src/java /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer/host /nutch/trunk/src/plugin/urlnormalizer-host/src/java/org/apache/nutch/net/urlnormalizer/host/HostURLNormalizer.java /nutch/trunk/src/plugin/urlnormalizer-host/src/test /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer/host /nutch/trunk/src/plugin/urlnormalizer-host/src/test/org/apache/nutch/net/urlnormalizer/host/TestHostURLNormalizer.java
          Hide
          Hudson added a comment -

          Integrated in nutch-trunk-maven #331 (See https://builds.apache.org/job/nutch-trunk-maven/331/)
          NUTCH-1319 adding test to accomodate HostURLNormalizer (Revision 1353884)

          Result = SUCCESS
          markus :
          Files :

          • /nutch/trunk/src/test/org/apache/nutch/net/TestURLNormalizers.java
          Show
          Hudson added a comment - Integrated in nutch-trunk-maven #331 (See https://builds.apache.org/job/nutch-trunk-maven/331/ ) NUTCH-1319 adding test to accomodate HostURLNormalizer (Revision 1353884) Result = SUCCESS markus : Files : /nutch/trunk/src/test/org/apache/nutch/net/TestURLNormalizers.java
          Markus Jelsma made changes -
          Link This issue duplicates NUTCH-737 [ NUTCH-737 ]
          Gavin made changes -
          Link This issue blocks NUTCH-1326 [ NUTCH-1326 ]
          Gavin made changes -
          Link This issue is depended upon by NUTCH-1326 [ NUTCH-1326 ]
          Lewis John McGibbney made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Tien Nguyen Manh made changes -
          Link This issue relates to NUTCH-1702 [ NUTCH-1702 ]
          Hide
          Markus Jelsma added a comment -

          need to edit

          Show
          Markus Jelsma added a comment - need to edit
          Markus Jelsma made changes -
          Resolution Fixed [ 1 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          Markus Jelsma made changes -
          Description Nutch would benefit from having a host normalizer. A host normalizer maps a given host to the desired host. A basic example is to map www.apache.org to apache.org. The Apache website is one of many on the internet that has a duplicate website on the same domain just because it allows both www and non-www to return HTTP 200 and proper content.

          It is also able to handle wildcards such as *.example.org to example.org if there are multiple sub domains that actually point to the same website.

          Large internet crawls tend to get polluted very quickly due to these problems. It also leads to skewed scores in the webgraph as different websites link to different versions of the same duplicate website.
          Nutch would benefit from having a host normalizer. A host normalizer maps a given host to the desired host. A basic example is to map www.apache.org to apache.org. The Apache website is one of many on the internet that has a duplicate website on the same domain just because it allows both www and non-www to return HTTP 200 and proper content.

          It is also able to handle wildcards such as *.example.org to example.org if there are multiple sub domains that actually point to the same website.

          Large internet crawls tend to get polluted very quickly due to these problems. It also leads to skewed scores in the webgraph as different websites link to different versions of the same duplicate website. An example:

          {code}
          # Force all sub domains to www.
          *.example.com example.com

          # Force no sub domain to www. URL's
          www.example.net example.net

          # Force www. sub domain when hitting link without sub domain
          example.org www.example.org
          {code}
          Hide
          Markus Jelsma added a comment -

          editted

          Show
          Markus Jelsma added a comment - editted
          Markus Jelsma made changes -
          Status Reopened [ 4 ] Closed [ 6 ]
          Resolution Fixed [ 1 ]
          Markus Jelsma made changes -
          Resolution Fixed [ 1 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          Markus Jelsma made changes -
          Description Nutch would benefit from having a host normalizer. A host normalizer maps a given host to the desired host. A basic example is to map www.apache.org to apache.org. The Apache website is one of many on the internet that has a duplicate website on the same domain just because it allows both www and non-www to return HTTP 200 and proper content.

          It is also able to handle wildcards such as *.example.org to example.org if there are multiple sub domains that actually point to the same website.

          Large internet crawls tend to get polluted very quickly due to these problems. It also leads to skewed scores in the webgraph as different websites link to different versions of the same duplicate website. An example:

          {code}
          # Force all sub domains to www.
          *.example.com example.com

          # Force no sub domain to www. URL's
          www.example.net example.net

          # Force www. sub domain when hitting link without sub domain
          example.org www.example.org
          {code}
          Nutch would benefit from having a host normalizer. A host normalizer maps a given host to the desired host. A basic example is to map www.apache.org to apache.org. The Apache website is one of many on the internet that has a duplicate website on the same domain just because it allows both www and non-www to return HTTP 200 and proper content.

          It is also able to handle wildcards such as *.example.org to example.org if there are multiple sub domains that actually point to the same website.

          Large internet crawls tend to get polluted very quickly due to these problems. It also leads to skewed scores in the webgraph as different websites link to different versions of the same duplicate website. An example:

          {code}
          # Force all sub domains to non-www.
          *.example.com example.com

          # Force www sub domain to non-www.
          www.example.net example.net

          # Force non-www. sub domain to www
          example.org www.example.org
          {code}
          Hide
          Markus Jelsma added a comment -

          Added incorrect example,. fixed!

          Show
          Markus Jelsma added a comment - Added incorrect example,. fixed!
          Markus Jelsma made changes -
          Status Reopened [ 4 ] Closed [ 6 ]
          Resolution Fixed [ 1 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          82d 13h 28m 1 Markus Jelsma 12/Jun/12 11:33
          Resolved Resolved Closed Closed
          343d 17h 20m 1 Lewis John McGibbney 22/May/13 04:53
          Closed Closed Reopened Reopened
          776d 8h 4m 2 Markus Jelsma 07/Jul/15 12:58
          Reopened Reopened Closed Closed
          1m 24s 2 Markus Jelsma 07/Jul/15 12:59

            People

            • Assignee:
              Markus Jelsma
              Reporter:
              Markus Jelsma
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development