Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-6110

FlowFile Repository can fail to update

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.10.0, 1.9.1
    • Component/s: Core Framework
    • Labels:
      None

      Description

      The following errors were reported on the mailing list:

       
      2019-03-06 14:33:20,305 ERROR [Timer-Driven Process Thread-47] o.a.n.c.r.StandardProcessSession Failed to commit session org.apache.nifi.processor.exception.ProcessException: FlowFile Repository failed to update. Will roll back.

      2019-03-06 14:33:20,306 ERROR [Timer-Driven Process Thread-47] o.a.n.processors.standard.ConvertRecord ConvertRecord[id=32781c92-fab3-34de-b1ff-6138f8788701] Failed to commit session StandardProcessSession[id=244416156] due to org.apache.nifi.processor.exception.ProcessException: FlowFile Repository failed to update; rolling back: org.apache.nifi.processor.exception.ProcessException: FlowFile Repository failed to update
      org.apache.nifi.processor.exception.ProcessException: FlowFile Repository failed to update
      at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:413)
      at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:343)
      at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:256)
      at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.nio.file.FileAlreadyExistsException: /flowfile_repo/nifi-1.9.0/journals/overflow-874061
      at sun.nio.fs.UnixException.translateToIOException(UnixException.java:88)
      at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
      at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
      at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
      at java.nio.file.Files.createDirectory(Files.java:674)
      at org.apache.nifi.wali.LengthDelimitedJournal.update(LengthDelimitedJournal.java:249)
      at org.apache.nifi.wali.SequentialAccessWriteAheadLog.update(SequentialAccessWriteAheadLog.java:123)
      at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.updateRepository(WriteAheadFlowFileRepository.java:309)
      at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.updateRepository(WriteAheadFlowFileRepository.java:259)
      at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:407)
      ... 10 common frames omitted
      2019-03-06 14:33:20,337 ERROR [Timer-Driven Process Thread-53] o.a.n.c.r.StandardProcessSession Failed to commit session org.apache.nifi.processor.exception.ProcessException: FlowFile Repository failed to update. Will roll back.
      2019-03-06 14:33:20,337 ERROR [Timer-Driven Process Thread-53] o.a.n.processors.standard.ConvertRecord ConvertRecord[id=32781c92-fab3-34de-b1ff-6138f8788701] Failed to commit session StandardProcessSession[id=244415981] due to org.apache.nifi.processor.exception.ProcessException: FlowFile Repository failed to update; rolling back: org.apache.nifi.processor.exception.ProcessException: FlowFile Repository failed to update
      org.apache.nifi.processor.exception.ProcessException: FlowFile Repository failed to update
      at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:413)
      at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:343)
      at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:256)
      at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: java.io.IOException: Stream Closed
      at java.io.FileOutputStream.writeBytes(Native Method)
      at java.io.FileOutputStream.write(FileOutputStream.java:326)
      at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
      at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
      at org.apache.nifi.wali.LengthDelimitedJournal.update(LengthDelimitedJournal.java:308)
      at org.apache.nifi.wali.SequentialAccessWriteAheadLog.update(SequentialAccessWriteAheadLog.java:123)
      at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.updateRepository(WriteAheadFlowFileRepository.java:309)
      at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.updateRepository(WriteAheadFlowFileRepository.java:259)
      at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:407)... 10 common frames omitted
      .......MANY errors
       
      After 26 rolls of the logfiles the checkpoint arrives
      2019-03-06 14:34:53,671 INFO [pool-13-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 22744 records in 339 milliseconds
       
      And then a new repository file fails
      2019-03-06 14:34:54,287 ERROR [Timer-Driven Process Thread-93] o.a.n.processors.standard.ConvertRecord ConvertRecord[id=32781c92-fab3-34de-b1ff-6138f8788701] Failed to commit session StandardProcessSession[id=246475219] due to org.apache.nifi.processor.exception.ProcessException: FlowFile Repository failed to update; rolling back: org.apache.nifi.processor.exception.ProcessException: FlowFile Repository failed to update

      org.apache.nifi.processor.exception.ProcessException: FlowFile Repository failed to update
      at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:413)
      at org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:343)
      at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:256)
      at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)Caused by: java.nio.file.FileAlreadyExistsException: /flowfile_repo/nifi-1.9.0/journals/overflow-876686
       
       
      The following also start showing up after a while.
      2019-03-06 14:33:22,363 ERROR [Timer-Driven Process Thread-19] o.a.n.c.r.c.StandardResourceClaimManager Decremented claimant count for StandardResourceClaim[id=[1551879190172-8371|tel:1551879190172-8371], container=default, section=179] to -3333

      2019-03-06 14:33:22,363 WARN [Timer-Driven Process Thread-53] o.a.n.c.r.c.StandardResourceClaimManager Decrementing claimant count for StandardResourceClaim[id=[1551879161479-8345|tel:1551879161479-8345], container=default, section=153] but claimant count is not known. Returning -1
      2019-03-06 14:33:22,363 WARN [Timer-Driven Process Thread-47] o.a.n.c.r.c.StandardResourceClaimManager Decrementing claimant count for StandardResourceClaim[id=[1551879161479-8346|tel:1551879161479-8346], container=default, section=154] but claimant count is not known. Returning -1
      2019-03-06 14:33:22,363 WARN [Timer-Driven Process Thread-47] o.a.n.c.r.c.StandardResourceClaimManager Decrementing claimant count for StandardResourceClaim[id=[1551879161479-8345|tel:1551879161479-8345], container=default, section=153] but claimant count is not known. Returning -12019-03-06 14:33:22,363 ERROR [Timer-Driven Process Thread-94] o.a.n.c.r.c.StandardResourceClaimManager Decremented claimant count for StandardResourceClaim[id=[1551879190172-8371|tel:1551879190172-8371], container=default, section=179] to -3334

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                markap14 Mark Payne
                Reporter:
                markap14 Mark Payne
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 40m
                  40m