Uploaded image for project: 'ManifoldCF'
  1. ManifoldCF
  2. CONNECTORS-1395

Unexpected jobqueue status - record id 1488898668325, expecting active status, saw 4

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: ManifoldCF 2.6
    • Fix Version/s: ManifoldCF 2.7
    • Component/s: Framework core
    • Labels:
      None

      Description

      User saw this in the log, after which the system hung:

      ERROR 2017-03-08 00:25:30,433 (Worker thread '14') - Exception tossed: Unexpected jobqueue status - record id 1488898668325, expecting active status, saw 4
      org.apache.manifoldcf.core.interfaces.ManifoldCFException: Unexpected jobqueue status - record id 1488898668325, expecting active status, saw 4
                      at org.apache.manifoldcf.crawler.jobs.JobQueue.updateCompletedRecord(JobQueue.java:1019)
                      at org.apache.manifoldcf.crawler.jobs.JobManager.markDocumentCompletedMultiple(JobManager.java:3271)
                      at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:710)
      WARN 2017-03-08 00:25:30,449 (Worker thread '23') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:30,449 (Worker thread '24') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:30,464 (Worker thread '9') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:30,464 (Worker thread '0') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:31,900 (Worker thread '11') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:31,900 (Worker thread '29') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:32,867 (Worker thread '10') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:32,867 (Worker thread '2') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:33,335 (Worker thread '8') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:36,642 (Worker thread '20') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:37,422 (Worker thread '21') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:38,280 (Worker thread '22') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:38,280 (Worker thread '3') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:38,280 (Worker thread '5') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:38,826 (Worker thread '28') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:39,045 (Worker thread '13') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:45,425 (Worker thread '4') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:45,425 (Worker thread '15') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:45,425 (Worker thread '17') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:46,392 (Worker thread '25') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:46,392 (Worker thread '27') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:26:11,043 (Worker thread '1') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:26:35,817 (Worker thread '19') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:26:35,817 (Worker thread '26') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:26:36,753 (Worker thread '7') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:26:39,248 (Worker thread '6') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:26:39,248 (Worker thread '18') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:26:43,129 (Worker thread '16') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      FATAL 2017-03-08 00:32:24,819 (Idle cleanup thread) - Error tossed: Can't release lock we don't hold
      java.lang.IllegalStateException: Can't release lock we don't hold”
      

        Attachments

        1. ZK Console Output.txt
          36 kB
          Guy Standen
        2. ZK Console 2.txt
          10 kB
          Guy Standen
        3. MCF Log 2.txt
          90 kB
          Guy Standen
        4. PostgreSQL Log Extract 4.txt
          4 kB
          Guy Standen
        5. Manifoldcf log4.txt
          10 kB
          Guy Standen
        6. WebApps Stack 4.txt
          19 kB
          Guy Standen
        7. Agents Stack 4.txt
          65 kB
          Guy Standen
        8. ZK log 4.txt
          551 kB
          Guy Standen
        9. MCF Log 5.txt
          45 kB
          Guy Standen

          Activity

            People

            • Assignee:
              kwright@metacarta.com Karl Wright
              Reporter:
              kwright@metacarta.com Karl Wright
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: