Details
-
Bug
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
2.0.0-M1
-
None
-
None
-
None
Description
I caused Xodus to timeout writing a transation to flow configuration history. Thereafter, every operation that would update flow configuration history, or trying to view flow configuration history, continues to throw an exception. The UI shows "An unexpected error has occurred. Please check the logs for additional details" based on a 500 response from the REST API /nifi-api/flow/history
I had to restart NiFi to recover. Hopefully, this code can be improved to detect exceptions and recover on its own.
I caused the initial problem by setting a breakpoint while a debugger was attached to NiFi. I dragged a new Process Group onto the graph from a flow definition file and let it sit at the breakpoint for a while.
2023-12-04 16:32:00,448 ERROR [NiFi Web Server-466] jetbrains.exodus.env.EnvironmentImpl Failed to flush transaction java.io.IOException: Stream Closed at java.base/java.io.RandomAccessFile.length0(Native Method) at java.base/java.io.RandomAccessFile.length(RandomAccessFile.java:653) at jetbrains.exodus.io.FileDataWriter.ensureFile(FileDataWriter.kt:172) at jetbrains.exodus.io.FileDataWriter.write(FileDataWriter.kt:59) at jetbrains.exodus.log.BufferedDataWriter.writePage(BufferedDataWriter.java:264) at jetbrains.exodus.log.BufferedDataWriter.flush(BufferedDataWriter.java:176) at jetbrains.exodus.log.Log.flush(Log.kt:631) at jetbrains.exodus.log.Log.flush$default(Log.kt:630) at jetbrains.exodus.log.Log.flush(Log.kt) at jetbrains.exodus.env.EnvironmentImpl.flushTransaction(EnvironmentImpl.java:731) at jetbrains.exodus.env.EnvironmentImpl.commitTransaction(EnvironmentImpl.java:684) at jetbrains.exodus.env.ReadWriteTransaction.commit(ReadWriteTransaction.java:96) at jetbrains.exodus.entitystore.PersistentStoreTransaction.doCommit(PersistentStoreTransaction.java:192) at jetbrains.exodus.entitystore.PersistentStoreTransaction.commit(PersistentStoreTransaction.java:180) at jetbrains.exodus.entitystore.PersistentEntityStoreImpl.executeInExclusiveTransaction(PersistentEntityStoreImpl.java:720) at org.apache.nifi.admin.service.EntityStoreAuditService.addActions(EntityStoreAuditService.java:116) at org.apache.nifi.audit.NiFiAuditor.saveActions(NiFiAuditor.java:66) at org.apache.nifi.audit.NiFiAuditor.saveAction(NiFiAuditor.java:53) at org.apache.nifi.audit.ProcessGroupAuditor.saveUpdateAction(ProcessGroupAuditor.java:359) at org.apache.nifi.audit.ProcessGroupAuditor.updateProcessGroupFlowAdvice(ProcessGroupAuditor.java:305) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
Afterwards, any attempt to read or write from Xodus produces an exception similar to this:
2023-12-04 16:32:53,111 WARN [NiFi Web Server-371] o.apache.nifi.audit.ProcessGroupAuditor Unable to record actions: jetbrains.exodus.ExodusException: Environment is inoperative at jetbrains.exodus.ExodusException.toExodusException(ExodusException.java:49) at jetbrains.exodus.env.EnvironmentImpl.checkIsOperative(EnvironmentImpl.java:1069) at jetbrains.exodus.env.EnvironmentImpl.beginTransaction(EnvironmentImpl.java:613) at jetbrains.exodus.env.EnvironmentImpl.beginExclusiveTransaction(EnvironmentImpl.java:264) at jetbrains.exodus.entitystore.PersistentStoreTransaction.<init>(PersistentStoreTransaction.java:130) at jetbrains.exodus.entitystore.PersistentEntityStoreImpl.beginExclusiveTransaction(PersistentEntityStoreImpl.java:539) at jetbrains.exodus.entitystore.PersistentEntityStoreImpl.executeInExclusiveTransaction(PersistentEntityStoreImpl.java:717) at org.apache.nifi.admin.service.EntityStoreAuditService.addActions(EntityStoreAuditService.java:116) at org.apache.nifi.audit.NiFiAuditor.saveActions(NiFiAuditor.java:66) at org.apache.nifi.audit.NiFiAuditor.saveAction(NiFiAuditor.java:53) at org.apache.nifi.audit.ProcessGroupAuditor.saveUpdateAction(ProcessGroupAuditor.java:359) at org.apache.nifi.audit.ProcessGroupAuditor.updateProcessGroupFlowAdvice(ProcessGroupAuditor.java:305) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ... ... Caused by: java.io.IOException: Stream Closed at java.base/java.io.RandomAccessFile.length0(Native Method) at java.base/java.io.RandomAccessFile.length(RandomAccessFile.java:653) at jetbrains.exodus.io.FileDataWriter.ensureFile(FileDataWriter.kt:172) at jetbrains.exodus.io.FileDataWriter.write(FileDataWriter.kt:59) at jetbrains.exodus.log.BufferedDataWriter.writePage(BufferedDataWriter.java:264) at jetbrains.exodus.log.BufferedDataWriter.flush(BufferedDataWriter.java:176) at jetbrains.exodus.log.Log.flush(Log.kt:631) at jetbrains.exodus.log.Log.flush$default(Log.kt:630) at jetbrains.exodus.log.Log.flush(Log.kt) at jetbrains.exodus.env.EnvironmentImpl.flushTransaction(EnvironmentImpl.java:731) at jetbrains.exodus.env.EnvironmentImpl.commitTransaction(EnvironmentImpl.java:684) at jetbrains.exodus.env.ReadWriteTransaction.commit(ReadWriteTransaction.java:96) at jetbrains.exodus.entitystore.PersistentStoreTransaction.doCommit(PersistentStoreTransaction.java:192) at jetbrains.exodus.entitystore.PersistentStoreTransaction.commit(PersistentStoreTransaction.java:180) at jetbrains.exodus.entitystore.PersistentEntityStoreImpl.executeInExclusiveTransaction(PersistentEntityStoreImpl.java:720) at org.apache.nifi.admin.service.EntityStoreAuditService.addActions(EntityStoreAuditService.java:116) at org.apache.nifi.audit.NiFiAuditor.saveActions(NiFiAuditor.java:66) at org.apache.nifi.audit.NiFiAuditor.saveAction(NiFiAuditor.java:53) at org.apache.nifi.audit.ProcessGroupAuditor.saveUpdateAction(ProcessGroupAuditor.java:359) at org.apache.nifi.audit.ProcessGroupAuditor.updateProcessGroupFlowAdvice(ProcessGroupAuditor.java:305) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)