Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Invalid
    • Affects Version/s: 1.4
    • Fix Version/s: None
    • Component/s: parser
    • Labels:
      None

      Description

      Hello,

      Im currently trying to crawl a site which uses hashtags in the urls. I dont seem to get any results and Im hoping im just overlooking something.
      Site structure is as follows:
      http://domain.com (landingpage)
      http://domain.com/#/page1
      http://domain.com/#/page1/subpage1
      http://domain.com/#/page2
      http://domain.com/#/page2/subpage1
      and so on.

      I've pointed nutch to http://domain.com as start url and in my filter i've placed all kind of rules.
      First i thought this would be sufficient:
      +http\://domain\.com\/#
      But then i realised that # is used for comments so i escaped it:
      +http\://domain\.com\/#

      Still no results. So i thought i could use the asterix for it:
      +http\://domain\.com\/*
      Still no luck.. So i started using various regex stuff but without success.

      I noticed the following messages in hadoop.log:
      INFO anchor.AnchorIndexingFilter - Anchor deduplication is: off
      Ive researched on this setting but i dont know for sure if this affects my problem in a way. This property is set to false in my configs.

      I dont know if this is even related to the situation above but maybe it helps.

      Any help is very much appreciated! I've tried googling the problem but i couldnt find documentation or anyone else with this problem.

      Many thanks in advance.

      With kind regard,
      Roberto Gardenier

        Activity

        Hide
        Roberto Gardenier added a comment -

        Markus Jelsma,

        I got notified that you have closed my jira ticket, chaning its resolution status to Invalid.
        I wonder why you have closed my ticket and marked it invalid as i did not commit any changes or solutions?

        With kind regards,
        Roberto Gardenier

        ----Oorspronkelijk bericht----
        Van: Markus Jelsma (JIRA) jira@apache.org
        Verzonden: dinsdag 1 mei 2012 13:40
        Aan: r.gardenier@simgroep.nl
        Onderwerp: [jira] [Closed] (NUTCH-1343) Crawl sites with hashtags in url

        [ https://issues.apache.org/jira/browse/NUTCH-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

        Markus Jelsma closed NUTCH-1343.
        --------------------------------

        Resolution: Invalid


        This message is automatically generated by JIRA.
        If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
        For more information on JIRA, see: http://www.atlassian.com/software/jira

        Show
        Roberto Gardenier added a comment - Markus Jelsma, I got notified that you have closed my jira ticket, chaning its resolution status to Invalid. I wonder why you have closed my ticket and marked it invalid as i did not commit any changes or solutions? With kind regards, Roberto Gardenier ---- Oorspronkelijk bericht ---- Van: Markus Jelsma (JIRA) jira@apache.org Verzonden: dinsdag 1 mei 2012 13:40 Aan: r.gardenier@simgroep.nl Onderwerp: [jira] [Closed] ( NUTCH-1343 ) Crawl sites with hashtags in url [ https://issues.apache.org/jira/browse/NUTCH-1343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Markus Jelsma closed NUTCH-1343 . -------------------------------- Resolution: Invalid – This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
        Hide
        Markus Jelsma added a comment -

        Questions should be asked on the mailing list, as you just did. Concrete bugs and changes can be filed in Jira. Please check the mailinglist for replies to your inquiry.

        Show
        Markus Jelsma added a comment - Questions should be asked on the mailing list, as you just did. Concrete bugs and changes can be filed in Jira. Please check the mailinglist for replies to your inquiry.
        Hide
        Roberto Gardenier added a comment -

        Thank you for your response. I will check the mailinglist for any possible reactions. Thank you very much.

        Show
        Roberto Gardenier added a comment - Thank you for your response. I will check the mailinglist for any possible reactions. Thank you very much.

          People

          • Assignee:
            Unassigned
            Reporter:
            Roberto Gardenier
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development