Uploaded image for project: 'ManifoldCF'
  1. ManifoldCF
  2. CONNECTORS-502

Crawling with MySQL as a backend shows signs of transactional integrity problems

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: ManifoldCF 0.6
    • Fix Version/s: ManifoldCF 1.0
    • Labels:
      None
    • Environment:

      MySQL 5.1

      Description

      When running the webcrawler-loadtests-mysql target, I got the following error:

      ERROR 2012-08-12 20:59:35,664 (Worker thread '10') - Exception tossed: Unexpected jobqueue status - record id 1344819115577, expecting active status, saw 0
      org.apache.manifoldcf.core.interfaces.ManifoldCFException: Unexpected jobqueue status - record id 1344819115577, expecting active status, saw 0
      	at org.apache.manifoldcf.crawler.jobs.JobQueue.updateOrHopcountRemoveRecord(JobQueue.java:800)
      	at org.apache.manifoldcf.crawler.jobs.JobManager.markDocumentHopcountRemovalMultiple(JobManager.java:2583)
      	at org.apache.manifoldcf.crawler.system.WorkerThread.processJobQueueHopcountRemovals(WorkerThread.java:1100)
      	at org.apache.manifoldcf.crawler.system.WorkerThread.processHopcountRemovalLists(WorkerThread.java:952)
      	at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:790)
      

      This indicates that although the stuffer thread thought it had put the document into the ACTIVE state, it had either never made it to the ACTIVE state or had been overridden by some other code. Either way, MySQL transactions do not seem to be working properly.

        Attachments

          Activity

            People

            • Assignee:
              kwright@metacarta.com Karl Wright
              Reporter:
              kwright@metacarta.com Karl Wright
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: