Droids
  1. Droids
  2. DROIDS-77

Be able to modify URL rules while crawler is running

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: core
    • Labels:
      None

      Description

      It would be nice to be able to modify the URL rules while a crawler is running. This would allow me to dynamically exclude areas from being crawled based on results being returned. Basically I want to look for certain markers inside a page, then not crawl those pages without having update a robots file. Different paths of our site is going to enter into the index from a different method than the main crawl, so I can skip them once I find them.

      Having a modifiable filter would allow people to load their rules from places other than a file without having to write their own implementation or extension. I'll try to work up a patch sometime this week.

        Issue Links

          Activity

          Richard Frovarp created issue -
          Thorsten Scherler made changes -
          Field Original Value New Value
          Affects Version/s 0.0.2 [ 12314984 ]
          Affects Version/s 0.0.1 [ 12313486 ]
          Thorsten Scherler made changes -
          Affects Version/s 0.0.2 [ 12314984 ]
          Richard Frovarp made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Duplicate [ 3 ]
          Richard Frovarp made changes -
          Link This issue is duplicated by DROIDS-111 [ DROIDS-111 ]
          Richard Frovarp made changes -
          Link This issue is duplicated by DROIDS-111 [ DROIDS-111 ]
          Richard Frovarp made changes -
          Link This issue duplicates DROIDS-111 [ DROIDS-111 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          302d 22h 57m 1 Richard Frovarp 14/Dec/10 22:16

            People

            • Assignee:
              Unassigned
              Reporter:
              Richard Frovarp
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development