Uploaded image for project: 'ManifoldCF'
  1. ManifoldCF
  2. CONNECTORS-1395

Unexpected jobqueue status - record id 1488898668325, expecting active status, saw 4

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • ManifoldCF 2.6
    • ManifoldCF 2.7
    • Framework core
    • None

    Description

      User saw this in the log, after which the system hung:

      ERROR 2017-03-08 00:25:30,433 (Worker thread '14') - Exception tossed: Unexpected jobqueue status - record id 1488898668325, expecting active status, saw 4
      org.apache.manifoldcf.core.interfaces.ManifoldCFException: Unexpected jobqueue status - record id 1488898668325, expecting active status, saw 4
                      at org.apache.manifoldcf.crawler.jobs.JobQueue.updateCompletedRecord(JobQueue.java:1019)
                      at org.apache.manifoldcf.crawler.jobs.JobManager.markDocumentCompletedMultiple(JobManager.java:3271)
                      at org.apache.manifoldcf.crawler.system.WorkerThread.run(WorkerThread.java:710)
      WARN 2017-03-08 00:25:30,449 (Worker thread '23') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:30,449 (Worker thread '24') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:30,464 (Worker thread '9') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:30,464 (Worker thread '0') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:31,900 (Worker thread '11') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:31,900 (Worker thread '29') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:32,867 (Worker thread '10') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:32,867 (Worker thread '2') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:33,335 (Worker thread '8') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:36,642 (Worker thread '20') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:37,422 (Worker thread '21') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:38,280 (Worker thread '22') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:38,280 (Worker thread '3') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:38,280 (Worker thread '5') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:38,826 (Worker thread '28') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:39,045 (Worker thread '13') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:45,425 (Worker thread '4') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:45,425 (Worker thread '15') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:45,425 (Worker thread '17') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:46,392 (Worker thread '25') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:25:46,392 (Worker thread '27') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:26:11,043 (Worker thread '1') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:26:35,817 (Worker thread '19') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:26:35,817 (Worker thread '26') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:26:36,753 (Worker thread '7') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:26:39,248 (Worker thread '6') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:26:39,248 (Worker thread '18') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      WARN 2017-03-08 00:26:43,129 (Worker thread '16') - Service interruption reported for job 1488898090224 connection 'web': Job no longer active
      FATAL 2017-03-08 00:32:24,819 (Idle cleanup thread) - Error tossed: Can't release lock we don't hold
      java.lang.IllegalStateException: Can't release lock we don't hold”
      

      Attachments

        1. ZK log 4.txt
          551 kB
          Guy Standen
        2. ZK Console Output.txt
          36 kB
          Guy Standen
        3. ZK Console 2.txt
          10 kB
          Guy Standen
        4. WebApps Stack 4.txt
          19 kB
          Guy Standen
        5. PostgreSQL Log Extract 4.txt
          4 kB
          Guy Standen
        6. MCF Log 5.txt
          45 kB
          Guy Standen
        7. MCF Log 2.txt
          90 kB
          Guy Standen
        8. Manifoldcf log4.txt
          10 kB
          Guy Standen
        9. Agents Stack 4.txt
          65 kB
          Guy Standen

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            kwright@metacarta.com Karl Wright
            kwright@metacarta.com Karl Wright
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment