Droids
  1. Droids
  2. DROIDS-146

MultiThreadedTaskMaster stops on HTTP error code 404

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.2.0
    • Fix Version/s: 0.2.0
    • Component/s: core
    • Labels:
      None

      Description

      Crawling a site and getting a HTTP error >= 400 the MultiThreadedTaskMaster stops the process.

       
      1:12:27.312 [pool-1-thread-1] ERROR org.apache.droids.AbstractDroid -
      org.apache.http.client.HttpResponseException: Not Found
              at org.apache.droids.protocol.http.HttpProtocol.load(HttpProtocol.java:71) ~[droids-core-0.2-incubating-SNAPSHOT.jar:0.2-incubating-SNAPSHOT 1146608 - truebner]
              at org.apache.droids.robot.crawler.CrawlingWorker.execute(CrawlingWorker.java:72) ~[droids-core-0.2-incubating-SNAPSHOT.jar:0.2-incubating-SNAPSHOT 1146608 - truebner]
              at org.apache.droids.robot.crawler.CrawlingWorker.execute(CrawlingWorker.java:39) ~[droids-core-0.2-incubating-SNAPSHOT.jar:0.2-incubating-SNAPSHOT 1146608 - truebner]
              at org.apache.droids.impl.MultiThreadedTaskMaster$TaskExecutor.run(MultiThreadedTaskMaster.java:335) ~[droids-core-0.2-incubating-SNAPSHOT.jar:0.2-incubating-SNAPSHOT 1146608 - truebner]
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_24]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_24]
              at java.lang.Thread.run(Thread.java:662) [na:1.6.0_24]
      
      1. DROIDS-146.patch
        2 kB
        Tobias Rübner

        Issue Links

          Activity

          Hide
          Tobias Rübner added a comment -

          duplicate of DROIDS-152

          Show
          Tobias Rübner added a comment - duplicate of DROIDS-152
          Hide
          Bertil Chapuis added a comment -

          Personally, I often implement a custom TaskExceptionHandler. One of the advantages of handling the different cases in the TaskExceptionHandler is that the exception may as well occur during the parsing process. In my opinion, including all these cases in the TaskMaster won't be manageable. However, providing more TaskExceptionHandler implementations which answers common use cases could be a really interesting way to solve your issue.

          Show
          Bertil Chapuis added a comment - Personally, I often implement a custom TaskExceptionHandler. One of the advantages of handling the different cases in the TaskExceptionHandler is that the exception may as well occur during the parsing process. In my opinion, including all these cases in the TaskMaster won't be manageable. However, providing more TaskExceptionHandler implementations which answers common use cases could be a really interesting way to solve your issue.
          Hide
          Tobias Rübner added a comment -

          Currently the ExceptionHandler org.apache.droids.impl.DefaultTaskExceptionHandler does only return a warning.
          But in the current code of the MultiThreadedTaskMaster each exception stops the process.

          Show
          Tobias Rübner added a comment - Currently the ExceptionHandler org.apache.droids.impl.DefaultTaskExceptionHandler does only return a warning. But in the current code of the MultiThreadedTaskMaster each exception stops the process.
          Hide
          Bertil Chapuis added a comment -

          Thanks for the patch. I may have miss something but what about handling this case with a custom ExceptionHanlder which do not return FATAL when a 404 error is encountered?

          Show
          Bertil Chapuis added a comment - Thanks for the patch. I may have miss something but what about handling this case with a custom ExceptionHanlder which do not return FATAL when a 404 error is encountered?

            People

            • Assignee:
              Unassigned
              Reporter:
              Tobias Rübner
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development