Details
-
Improvement
-
Status: In Progress
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
When running oak-run revisions detailedGC command, if the cleanup is too big the processing time can exceed the Mongo cursor timeout, which defaults to 30 minutes:
java.io.IOException: com.mongodb.MongoCursorNotFoundException: Query failed with error code -5 and error message 'Cursor 1007212049743349339 not found on server mongohost' on server mongohost:27017 at org.apache.jackrabbit.oak.run.RevisionsCommand.collectOnce(RevisionsCommand.java:427) at org.apache.jackrabbit.oak.run.RevisionsCommand.collect(RevisionsCommand.java:377) at org.apache.jackrabbit.oak.run.RevisionsCommand.execute(RevisionsCommand.java:245) at org.apache.jackrabbit.oak.run.Main.main(Main.java:49) Caused by: com.mongodb.MongoCursorNotFoundException: Query failed with error code -5 and error message 'Cursor 1007212049743349339 not found on server mongohost:27017' on server mongohost:27017 at com.mongodb.operation.QueryHelper.translateCommandException(QueryHelper.java:27) at com.mongodb.operation.QueryBatchCursor.getMore(QueryBatchCursor.java:268) at com.mongodb.operation.QueryBatchCursor.hasNext(QueryBatchCursor.java:139) at com.mongodb.client.internal.MongoBatchCursorAdapter.hasNext(MongoBatchCursorAdapter.java:54) at org.apache.jackrabbit.guava.common.collect.TransformedIterator.hasNext(TransformedIterator.java:46) at org.apache.jackrabbit.guava.common.collect.Iterators$5.computeNext(Iterators.java:670) at org.apache.jackrabbit.guava.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:145) at org.apache.jackrabbit.guava.common.collect.AbstractIterator.hasNext(AbstractIterator.java:140) at org.apache.jackrabbit.guava.common.collect.Iterators$ConcatenatedIterator.hasNext(Iterators.java:1395) at org.apache.jackrabbit.oak.plugins.document.SplitDocumentCleanUp.disconnect(SplitDocumentCleanUp.java:63) at org.apache.jackrabbit.oak.plugins.document.VersionGCSupport.deleteSplitDocuments(VersionGCSupport.java:146) at org.apache.jackrabbit.oak.plugins.document.VersionGarbageCollector$GCJob.collectSplitDocuments(VersionGarbageCollector.java:842) at org.apache.jackrabbit.oak.plugins.document.VersionGarbageCollector$GCJob.gc(VersionGarbageCollector.java:686) at org.apache.jackrabbit.oak.plugins.document.VersionGarbageCollector$GCJob.run(VersionGarbageCollector.java:653) at org.apache.jackrabbit.oak.plugins.document.VersionGarbageCollector.gc(VersionGarbageCollector.java:221) at org.apache.jackrabbit.oak.run.RevisionsCommand$2.call(RevisionsCommand.java:401) at org.apache.jackrabbit.oak.run.RevisionsCommand$2.call(RevisionsCommand.java:398) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) 15:45:08.184 [DocumentNodeStore cluster update thread (0)] WARN o.a.j.o.p.document.DocumentNodeStore - Background operation failed: java.lang.reflect.UndeclaredThrowableException java.lang.reflect.UndeclaredThrowableException: null at com.sun.proxy.$Proxy0.query(Unknown Source) at org.apache.jackrabbit.oak.plugins.document.ClusterNodeInfoDocument.all(ClusterNodeInfoDocument.java:154) at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.updateClusterState(DocumentNodeStore.java:2480) at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$BackgroundClusterUpdate.execute(DocumentNodeStore.java:3931) at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore$NodeStoreTask.run(DocumentNodeStore.java:3734) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.lang.reflect.InvocationTargetException: null at jdk.internal.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.jackrabbit.oak.plugins.document.util.ReadOnlyDocumentStoreWrapperFactory$1.invoke(ReadOnlyDocumentStoreWrapperFactory.java:41) ... 6 common frames omitted Caused by: java.lang.IllegalStateException: The pool is closed at com.mongodb.internal.connection.ConcurrentPool.get(ConcurrentPool.java:137) at com.mongodb.internal.connection.ConcurrentPool.get(ConcurrentPool.java:123) at com.mongodb.internal.session.ServerSessionPool.get(ServerSessionPool.java:80) at com.mongodb.internal.session.BaseClientSessionImpl.<init>(BaseClientSessionImpl.java:44) at com.mongodb.client.internal.ClientSessionImpl.<init>(ClientSessionImpl.java:56) at com.mongodb.client.internal.MongoClientDelegate.createClientSession(MongoClientDelegate.java:120) at com.mongodb.Mongo.createClientSession(Mongo.java:837) at com.mongodb.MongoClient.startSession(MongoClient.java:582) at org.apache.jackrabbit.oak.plugins.document.mongo.MongoSessionFactory.createClientSession(MongoSessionFactory.java:55) at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDBConnection.createClientSession(MongoDBConnection.java:115) at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.createClientSession(MongoDocumentStore.java:2405) at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.execute(MongoDocumentStore.java:2389) at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.queryInternal(MongoDocumentStore.java:846) at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.queryWithRetry(MongoDocumentStore.java:774) at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.query(MongoDocumentStore.java:751) at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.query(MongoDocumentStore.java:739) at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.query(MongoDocumentStore.java:728) ... 10 common frames omitted
Check if this needs mitigation or could happen as part of the internal repository detailedGC execution.
Attachments
Issue Links
- links to