Picked up JAVA_TOOL_OPTIONS: -javaagent:/elastic/apm/agent/elastic-apm-agent.jar 2022-07-15 04:16:31,947 [main] INFO co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean 2022-07-15 04:16:32,045 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.23.0 as saman-recovery-temp on Java 1.8.0_212 Runtime version: 1.8.0_212-b04 VM version: 25.212-b04 (IcedTea) Linux 3.10.0-1062.1.2.el7.x86_64 2022-07-15 04:16:32,046 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - VM Arguments: [-javaagent:/elastic/apm/agent/elastic-apm-agent.jar, -Djava.security.egd=file:/dev/./urandom, -Djava.awt.headless=true, -Xmx6144m, -Xms4096m, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath="/gam/temp"] 2022-07-15 04:16:32,047 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - service_name: 'saman-recovery-temp' (source: Environment Variables) 2022-07-15 04:16:32,047 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - environment: 'production' (source: Environment Variables) 2022-07-15 04:16:32,047 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - server_url: 'http://apm-server.dci.ir' (source: Environment Variables) 2022-07-15 04:16:32,047 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - application_packages: 'com.gam.phoenix.samanapi' (source: Environment Variables) 2022-07-15 04:16:32,047 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - log_level: 'INFO' (source: Environment Variables) 2022-07-15 04:16:35,220 [elastic-apm-metadata] WARN co.elastic.apm.agent.impl.CloudMetadataProvider - Unexpected error during automatic discovery process for cloud provider java.util.concurrent.TimeoutException: null at java.util.concurrent.FutureTask.get(FutureTask.java:205) ~[?:1.8.0_212] at co.elastic.apm.agent.impl.CloudMetadataProvider.tryAllCloudProviders(CloudMetadataProvider.java:165) [?:?] at co.elastic.apm.agent.impl.CloudMetadataProvider.fetchAndParseCloudProviderInfo(CloudMetadataProvider.java:107) [?:?] at co.elastic.apm.agent.impl.MetaData$1.call(MetaData.java:125) [?:?] at co.elastic.apm.agent.impl.MetaData$1.call(MetaData.java:121) [?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212] 2022-07-15 04:16:35,690 [main] INFO co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state 2022-07-15 04:16:35,836 [elastic-apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: { "build_date": "2020-08-11T18:53:16Z", "build_sha": "41f41134f4dfe91966765f5a712c7302e71bc56d", "version": "7.9.0"} 04:16:37,345 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@dc9876b - URL [jar:file:/saman-api-core.jar!/BOOT-INF/classes!/logback-spring.xml] is not of type file 04:16:37,565 |-INFO in ch.qos.logback.core.joran.util.ConfigurationWatchListUtil@2f666ebb - Adding [jar:file:/saman-api-core.jar!/BOOT-INF/classes!/ecs.xml] to configuration watch list. 04:16:37,566 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@dc9876b - URL [jar:file:/saman-api-core.jar!/BOOT-INF/classes!/ecs.xml] is not of type file 04:16:37,569 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender] 04:16:37,573 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [ECS_METHOD_APPENDER] 04:16:37,610 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@1270836494 - setting totalSizeCap to 0 Bytes 04:16:37,614 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@1270836494 - Archive files will be limited to [10 MB] each. 04:16:37,631 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@1270836494 - Will use gz compression 04:16:37,633 |-INFO in c.q.l.core.rolling.SizeAndTimeBasedRollingPolicy@1270836494 - Will use the pattern LOG_FILE_IS_UNDEFINED.methods.json.%d{yyyy-MM-dd}.%i for the active file 04:16:37,636 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@30b6ffe0 - The date pattern is 'yyyy-MM-dd' from file name pattern 'LOG_FILE_IS_UNDEFINED.methods.json.%d{yyyy-MM-dd}.%i.gz'. 04:16:37,636 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@30b6ffe0 - Roll-over at midnight. 04:16:37,643 |-INFO in ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP@30b6ffe0 - Setting initial period to Fri Jul 15 04:16:37 IRDT 2022 04:16:37,647 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[ECS_METHOD_APPENDER] - Active log file name: spring-boot-logs/methods/saman-recovery-temp-59f8b4f4c-qdsxg.json 04:16:37,647 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[ECS_METHOD_APPENDER] - File property is set to [spring-boot-logs/methods/saman-recovery-temp-59f8b4f4c-qdsxg.json] 04:16:37,649 |-INFO in ch.qos.logback.core.joran.action.ConversionRuleAction - registering conversion word clr with class [org.springframework.boot.logging.logback.ColorConverter] 04:16:37,649 |-INFO in ch.qos.logback.core.joran.action.ConversionRuleAction - registering conversion word wex with class [org.springframework.boot.logging.logback.WhitespaceThrowableProxyConverter] 04:16:37,649 |-INFO in ch.qos.logback.core.joran.action.ConversionRuleAction - registering conversion word wEx with class [org.springframework.boot.logging.logback.ExtendedWhitespaceThrowableProxyConverter] 04:16:37,649 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender] 04:16:37,649 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [CONSOLE] 04:16:37,649 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property 04:16:37,680 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [com.gam.phoenix.spring.commons.logging.LoggingAdvice] to INFO 04:16:37,680 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@16c069df - Propagating INFO level on Logger[com.gam.phoenix.spring.commons.logging.LoggingAdvice] onto the JUL framework 04:16:37,681 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [com.gam.phoenix.spring.commons.logging.LoggingAdvice] to false 04:16:37,681 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [ECS_METHOD_APPENDER] to Logger[com.gam.phoenix.spring.commons.logging.LoggingAdvice] 04:16:37,681 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO 04:16:37,681 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@16c069df - Propagating INFO level on Logger[ROOT] onto the JUL framework 04:16:37,681 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [CONSOLE] to Logger[ROOT] 04:16:37,681 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration. 04:16:37,681 |-INFO in org.springframework.boot.logging.logback.SpringBootJoranConfigurator@26adfd2d - Registering current configuration as safe fallback point 04:16:37,739 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@16c069df - Propagating WARN level on Logger[com.gam.phoenix.samanapi.core.job.ResourceDateFolderJob] onto the JUL framework 04:16:37,740 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@16c069df - Propagating DEBUG level on Logger[org.apache.jackrabbit.oak.plugins.index.solr] onto the JUL framework 04:16:37,740 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@16c069df - Propagating DEBUG level on Logger[org.apache.jackrabbit.oak.query] onto the JUL framework 04:16:37,740 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@16c069df - Propagating DEBUG level on Logger[com.gam.phoenix.samanapi.core.repository] onto the JUL framework 04:16:37,740 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@16c069df - Propagating INFO level on Logger[org.apache.jackrabbit.oak.plugins.document] onto the JUL framework 04:16:37,740 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@16c069df - Propagating INFO level on Logger[org.apache.jackrabbit.oak.plugins.commit] onto the JUL framework 04:16:37,740 |-INFO in ch.qos.logback.classic.jul.LevelChangePropagator@16c069df - Propagating INFO level on Logger[org.apache.solr.client] onto the JUL framework 2022-07-15 04:16:37.949 --- [kground-preinit] : HV000001: Hibernate Validator 6.2.0.Final 2022-07-15 04:16:38.012 --- [ main] : Starting SamanApiApplication v3.5.2-SNAPSHOT using Java 1.8.0_212 on saman-recovery-temp-59f8b4f4c-qdsxg with PID 1 (/saman-api-core.jar started by root in /) 2022-07-15 04:16:38.012 --- [ main] : No active profile set, falling back to default profiles: default 2022-07-15 04:16:42.901 --- [ main] : Tomcat initialized with port(s): 8080 (http) 2022-07-15 04:16:42.930 --- [ main] : Initializing ProtocolHandler ["http-nio-8080"] 2022-07-15 04:16:42.930 --- [ main] : Starting service [Tomcat] 2022-07-15 04:16:42.931 --- [ main] : Starting Servlet engine: [Apache Tomcat/9.0.52] 2022-07-15 04:16:43.077 --- [ main] : Initializing Spring embedded WebApplicationContext 2022-07-15 04:16:43.077 --- [ main] : Root WebApplicationContext: initialization completed in 4954 ms 2022-07-15 04:16:43,205 [main] INFO co.elastic.apm.agent.micrometer.MicrometerMetricsReporter - Registering Micrometer MeterRegistry: io.micrometer.core.instrument.simple.SimpleMeterRegistry@64f555e7 2022-07-15 04:16:43.546 --- [ main] : Registered '/actuator/jolokia' to jolokia-actuator-endpoint 2022-07-15 04:16:44.770 --- [ main] : HikariPool-1 - Starting... 2022-07-15 04:16:45.516 --- [ main] : HikariPool-1 - Start completed. 2022-07-15 04:16:46.066 --- [ main] : RDBBlobStore (1.36.10-gam-recover-test) instantiated for database Oracle Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production (12.2), using driver: Oracle JDBC driver 12.2.0.1.0 (12.2), connecting to: jdbc:oracle:thin:@(DESCRIPTION= (SDU=32768)(ADDRESS=(PROTOCOL=TCP)(HOST=dms-rac-scan.mcci.local)(PORT=1521)) (ADDRESS=(PROTOCOL=TCP)(HOST=dms-rac-scan.mcci.local)(PORT=1521)) (LOAD_BALANCE=yes)(FAILOVER=ON)(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=dmslive)(failover_mode=(type=select)(method=basic)(retries=5)(delay=1)))), properties: {NLS_CHARACTERSET=AL32UTF8, NLS_COMP=BINARY}, transaction isolation level: TRANSACTION_READ_COMMITTED (2), {DATASTORE_DATA=.: ID VARCHAR2(64), DATA BLOB(-1) /* {BLOB=2004, VARCHAR2=12} */, DATASTORE_META=.: ID VARCHAR2(64), LVL NUMBER, LASTMOD NUMBER /* {NUMBER=2, VARCHAR2=12} */} 2022-07-15 04:16:46.066 --- [ main] : Tables present upon startup: [DATASTORE_DATA, DATASTORE_META] 2022-07-15 04:16:46.297 --- [ main] : RDBDocumentStore (1.36.10-gam-recover-test) instantiated for database Oracle Oracle Database 12c Enterprise Edition Release 12.2.0.1.0 - 64bit Production (12.2), using driver: Oracle JDBC driver 12.2.0.1.0 (12.2), connecting to: jdbc:oracle:thin:@(DESCRIPTION= (SDU=32768)(ADDRESS=(PROTOCOL=TCP)(HOST=dms-rac-scan.mcci.local)(PORT=1521)) (ADDRESS=(PROTOCOL=TCP)(HOST=dms-rac-scan.mcci.local)(PORT=1521)) (LOAD_BALANCE=yes)(FAILOVER=ON)(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=dmslive)(failover_mode=(type=select)(method=basic)(retries=5)(delay=1)))), properties: {NLS_CHARACTERSET=AL32UTF8, NLS_COMP=BINARY}, transaction isolation level: TRANSACTION_READ_COMMITTED (2), .: ID VARCHAR2(4000), MODIFIED NUMBER, HASBINARY NUMBER, DELETEDONCE NUMBER, MODCOUNT NUMBER, CMODCOUNT NUMBER, DSIZE NUMBER, VERSION NUMBER, SDTYPE NUMBER, SDMAXREVTIME NUMBER, DATA VARCHAR2(4000), BDATA BLOB(-1) /* {BLOB=2004, NUMBER=2, VARCHAR2=12} */ /* index NODES_MOD on GAM_SAMAN.NODES (MODIFIED) clustered (#4309504, p18404246), index NODES_SDM on GAM_SAMAN.NODES (SDMAXREVTIME) clustered (#126136, p1139), index NODES_SDT on GAM_SAMAN.NODES (SDTYPE) clustered (#4, p1303), index NODES_VSN on GAM_SAMAN.NODES (VERSION) clustered (#1, p5240107), unique index SYS_C0056952 on GAM_SAMAN.NODES (ID) clustered (#1657470444, p47402127) */ 2022-07-15 04:16:46.297 --- [ main] : Tables present upon startup: [CLUSTERNODES, NODES, SETTINGS, JOURNAL] 2022-07-15 04:16:46.661 --- [ main] : Acquired (existing) clusterId 56. MachineId (changed) mac:361dd6d22f48 -> mac:8e1b5fd0a73d, InstanceId / 2022-07-15 04:16:46.910 --- [ main] : ChangeSetBuilder enabled and size set to maxItems: 50, maxDepth: 9 2022-07-15 04:16:46.911 --- [ main] : Initialized DocumentNodeStore with clusterNodeId: 56, updateLimit: 100000 (id: 56, startTime: 1657842406590, machineId: mac:8e1b5fd0a73d, instanceId: /, pid: 1, uuid: 4cd3a8d2-20ab-45b7-95e5-2175d9edb5b7, readWriteMode: null, leaseCheckMode: STRICT, state: ACTIVE, oakVersion: 1.36.10-gam-recover-test, formatVersion: 1.8.0, invisible: false) 2022-07-15 04:16:47.061 --- [ main] : Bundling of nodes enabled 2022-07-15 04:16:48.745 --- [ main] : ACL created: 2022-07-15 04:16:48.765 --- [ main] : Loading CND files from /cnd-files folder 2022-07-15 04:16:48.771 --- [ main] : CND root folder abstract path is /cnd-files 2022-07-15 04:16:48.772 --- [ main] : Loading CND files from /cnd-files folder ended 2022-07-15 04:16:48.950 --- [ main] : Node and index "/oak:index/solr" created 2022-07-15 04:16:49.847 --- [ main] : org.xerial's sqlite-jdbc is not loaded. Please provide the jar on your classpath to parse sqlite files. See tika-parsers/pom.xml for the correct version. 2022-07-15 04:16:55.613 --- [ main] : Exposing 14 endpoint(s) beneath base path '/actuator' 2022-07-15 04:16:55.771 --- [ main] : Starting ProtocolHandler ["http-nio-8080"] 2022-07-15 04:16:55.797 --- [ main] : Tomcat started on port(s): 8080 (http) with context path '/saman/api/v1' 2022-07-15 04:16:55.824 --- [ main] : Started SamanApiApplication in 19.309 seconds (JVM running for 24.702) 2022-07-15 04:16:58,816 [http-nio-8080-exec-1] INFO co.elastic.apm.agent.servlet.ServletVersionInstrumentation - Servlet container info = Apache Tomcat/9.0.52 2022-07-15 04:16:58.819 --- [nio-8080-exec-1] : Initializing Spring DispatcherServlet 'dispatcherServlet' 2022-07-15 04:16:58.819 --- [nio-8080-exec-1] : Initializing Servlet 'dispatcherServlet' 2022-07-15 04:16:58.821 --- [nio-8080-exec-1] : Completed initialization in 2 ms 2022-07-15 04:18:06.592 --- [io-8080-exec-10] : Acquired recovery lock for cluster id 53 2022-07-15 04:18:06.592 --- [io-8080-exec-10] : Recovering candidates modified after: [2022-06-06 14:41:25.743] for clusterId [53] [sweepRev: r181387f73af-0-35] 2022-07-15 04:18:06.596 --- [io-8080-exec-10] : Performing Last Revision Recovery for clusterNodeId 53 2022-07-15 04:19:06.814 --- [io-8080-exec-10] : Recovery for cluster node [53]: 419780 nodes scanned in 60.21 s (~418301/m) - last interval 419780 nodes in 60.21 s (~418301/m) 2022-07-15 04:19:06.815 --- [io-8080-exec-10] : path: /oak:index/reference/:references/799417de-7e14-4823-834c-6fd9ca5eada7/contents/documents/dci/transactions/office-transactions/2/2121100/4201500022/1401/03/16/16/register/z5TXCDU2ILcgFRuRtvSj, unsaved: 0, unsavedParents: 425347, knownLastRevOrModification: 419303 2022-07-15 04:19:31,781 [http-nio-8080-exec-10] WARN co.elastic.apm.agent.impl.transaction.Span - Max spans (500) for transaction 'RepositoryController#recoverLastRevision' 00-ed659d532656d51e9b87f5772b031c04-79c05813bbf21dae-01 (7a35462f) has been reached. For this transaction and possibly others, further spans will be dropped. See config param 'transaction_max_spans'. 2022-07-15 04:20:06.809 --- [io-8080-exec-10] : Recovery for cluster node [53]: 823730 nodes scanned in 2.004 min (~411138/m) - last interval 403950 nodes in 60.00 s (~403950/m) 2022-07-15 04:20:06.810 --- [io-8080-exec-10] : path: /oak:index/uuid/:index/67ba3920-dcbe-48a4-a0bf-e5abd162e8be, unsaved: 0, unsavedParents: 790192, knownLastRevOrModification: 822863 2022-07-15 04:21:06.808 --- [io-8080-exec-10] : Recovery for cluster node [53]: 1226932 nodes scanned in 3.004 min (~408496/m) - last interval 403202 nodes in 60.00 s (~403202/m) 2022-07-15 04:21:06.808 --- [io-8080-exec-10] : path: /oak:index/reference/:references/0428cbfe-7029-4f0f-b9b5-73a916a309a0/contents/documents/dci/transactions/office-transactions/18/1181100/6101401000/1401/03/16/14/transfer/ooHRUpFRfyi7C0hXTeSV, unsaved: 0, unsavedParents: 1133967, knownLastRevOrModification: 1225790 2022-07-15 04:22:06.809 --- [io-8080-exec-10] : Recovery for cluster node [53]: 1554140 nodes scanned in 4.004 min (~388192/m) - last interval 327208 nodes in 60.00 s (~327208/m) 2022-07-15 04:22:06.809 --- [io-8080-exec-10] : path: /oak:index/nodetype/:index/dci%3Asubscription/contents/documents/dci/subscriptions/9/9/6/2/0/6/3/3/0/8/9962063308, unsaved: 0, unsavedParents: 1404960, knownLastRevOrModification: 1552738 2022-07-15 04:23:06.808 --- [io-8080-exec-10] : Recovery for cluster node [53]: 1920293 nodes scanned in 5.004 min (~383787/m) - last interval 366153 nodes in 60.00 s (~366153/m) 2022-07-15 04:23:06.809 --- [io-8080-exec-10] : path: /oak:index/reference/:references/866b4028-1a02-44c9-8fc2-ffe86e7309c1/contents/documents/dci/transactions/office-transactions/9/2191125/1109500003/1401/03/16/17/register/ZWF9U6HlfxTJCdhxcWSn/dci:resources, unsaved: 0, unsavedParents: 1703761, knownLastRevOrModification: 1918621 2022-07-15 04:24:06.808 --- [io-8080-exec-10] : Recovery for cluster node [53]: 2264220 nodes scanned in 6.004 min (~377147/m) - last interval 343927 nodes in 60.00 s (~343927/m) 2022-07-15 04:24:06.809 --- [io-8080-exec-10] : path: /oak:index/reference/:references/ac14cd95-5f4e-4733-89aa-3e59be9e8b5c/contents/documents/dci/transactions/office-transactions/9/2191100/1101700055/1401, unsaved: 0, unsavedParents: 1978285, knownLastRevOrModification: 2262353 2022-07-15 04:24:31,786 [http-nio-8080-exec-10] WARN co.elastic.apm.agent.impl.transaction.Span - Max spans (500) for transaction 'RepositoryController#recoverLastRevision' 00-ed659d532656d51e9b87f5772b031c04-79c05813bbf21dae-01 (7a35462f) has been reached. For this transaction and possibly others, further spans will be dropped. See config param 'transaction_max_spans'. 2022-07-15 04:25:06.808 --- [io-8080-exec-10] : Recovery for cluster node [53]: 2651120 nodes scanned in 7.004 min (~378540/m) - last interval 386900 nodes in 60.00 s (~386900/m) 2022-07-15 04:25:06.809 --- [io-8080-exec-10] : path: /oak:index/uuid/:index/ae0be1f8-8848-4616-85ea-8cef02105eb1, unsaved: 0, unsavedParents: 2281537, knownLastRevOrModification: 2648993 2022-07-15 04:26:06.812 --- [io-8080-exec-10] : Recovery for cluster node [53]: 3078650 nodes scanned in 8.004 min (~384658/m) - last interval 427530 nodes in 60.00 s (~427508/m) 2022-07-15 04:26:06.812 --- [io-8080-exec-10] : path: /oak:index/reference/:references/a1d4b00a-09d4-4f19-9548-af2e756e03b3/contents/documents/dci/transactions/office-transactions/8/2181100/3101801005/1401/03/16/23/register/VDcKinT7I3udFxtpUlS0, unsaved: 0, unsavedParents: 2609854, knownLastRevOrModification: 3076329 2022-07-15 04:27:06.811 --- [io-8080-exec-10] : Recovery for cluster node [53]: 3494480 nodes scanned in 9.004 min (~388121/m) - last interval 415830 nodes in 60.00 s (~415830/m) 2022-07-15 04:27:06.812 --- [io-8080-exec-10] : path: /oak:index/reference/:references/f3c5c3bb-6e2b-4a47-8df8-e72e5c597115/contents/documents, unsaved: 0, unsavedParents: 2919091, knownLastRevOrModification: 3492019 2022-07-15 04:28:06.812 --- [io-8080-exec-10] : Recovery for cluster node [53]: 3938830 nodes scanned in 10.00 min (~393741/m) - last interval 444350 nodes in 60.00 s (~444342/m) 2022-07-15 04:28:06.813 --- [io-8080-exec-10] : path: /contents/documents/dci/subscriptions/9/9/6/2/3/8/3/0/4/6/9962383046, unsaved: 0, unsavedParents: 3260401, knownLastRevOrModification: 3932533 2022-07-15 04:28:39.082 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8301ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:29:06.813 --- [io-8080-exec-10] : Recovery for cluster node [53]: 4310690 nodes scanned in 11.00 min (~391752/m) - last interval 371860 nodes in 60.00 s (~371853/m) 2022-07-15 04:29:06.814 --- [io-8080-exec-10] : path: /oak:index/reference/:references/b5a6556a-c757-449e-b00b-ed63263704be/contents/documents/dci/transactions/office-transactions/16/1161134/5408500002/1401/03/17/14/register/nWcdFMtVUWseHliXfASg, unsaved: 0, unsavedParents: 3542999, knownLastRevOrModification: 4304218 2022-07-15 04:29:31,799 [http-nio-8080-exec-10] WARN co.elastic.apm.agent.impl.transaction.Span - Max spans (500) for transaction 'RepositoryController#recoverLastRevision' 00-ed659d532656d51e9b87f5772b031c04-79c05813bbf21dae-01 (7a35462f) has been reached. For this transaction and possibly others, further spans will be dropped. See config param 'transaction_max_spans'. 2022-07-15 04:30:06.813 --- [io-8080-exec-10] : Recovery for cluster node [53]: 4780070 nodes scanned in 12.00 min (~398219/m) - last interval 469380 nodes in 60.00 s (~469380/m) 2022-07-15 04:30:06.814 --- [io-8080-exec-10] : path: /oak:index/nodetype/:index/gam%3AresourceReferenceMetadata/contents/documents/dci/transactions/office-transactions/4/2141128/3309500004/1401/03/17/15/register/LBFVU0CXH6i8flI7tRhM/dci:resources, unsaved: 0, unsavedParents: 3909575, knownLastRevOrModification: 4773328 2022-07-15 04:31:06.883 --- [io-8080-exec-10] : Recovery for cluster node [53]: 5233110 nodes scanned in 13.00 min (~402398/m) - last interval 453040 nodes in 60.07 s (~452512/m) 2022-07-15 04:31:06.883 --- [io-8080-exec-10] : path: /contents/images/temp/1401/03/17/11/30/0d740bc7-2f8c-487e-a6ea-9b06e99c047e-91000476562001jpg, unsaved: 0, unsavedParents: 4260954, knownLastRevOrModification: 5226128 2022-07-15 04:31:56.481 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 12675ms (of which the last update operation took 1ms) (expected ~1000ms) 2022-07-15 04:32:22.062 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 16567ms (expected ~1000ms) 2022-07-15 04:32:22.063 --- [io-8080-exec-10] : Recovery for cluster node [53]: 5609280 nodes scanned in 14.26 min (~393419/m) - last interval 376170 nodes in 75.18 s (~300223/m) 2022-07-15 04:32:22.064 --- [io-8080-exec-10] : path: /contents/documents/dci/subscriptions/9/1/7/7/6/9/1/0, unsaved: 0, unsavedParents: 4556060, knownLastRevOrModification: 5602108 2022-07-15 04:32:42.085 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 12014ms (expected ~1000ms) 2022-07-15 04:33:05.941 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 16847ms (expected ~1000ms) 2022-07-15 04:33:29.958 --- [io-8080-exec-10] : Recovery for cluster node [53]: 5798537 nodes scanned in 15.39 min (~376789/m) - last interval 189257 nodes in 67.89 s (~167252/m) 2022-07-15 04:33:29.959 --- [io-8080-exec-10] : path: /oak:index/nodetype/:index/gam%3AresourceReferenceMetadataContainer/contents/documents/dci/transactions/office-transactions/4380/2101100/2101500087/1401/03/17/15, unsaved: 0, unsavedParents: 4699234, knownLastRevOrModification: 5791275 2022-07-15 04:33:29.957 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 17003ms (expected ~1000ms) 2022-07-15 04:33:52.217 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 16248ms (expected ~1000ms) 2022-07-15 04:34:16.795 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 18562ms (expected ~1000ms) 2022-07-15 04:34:38.640 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 15834ms (expected ~1000ms) 2022-07-15 04:34:38.642 --- [io-8080-exec-10] : Recovery for cluster node [53]: 5962740 nodes scanned in 16.53 min (~360632/m) - last interval 164203 nodes in 68.69 s (~143435/m) 2022-07-15 04:34:38.642 --- [io-8080-exec-10] : path: /contents/images/temp/1401/03/17/12/41/f2e9ed3e-f0ef-44fb-adaa-d7e97eceb3e3-550012jpg, unsaved: 0, unsavedParents: 4824530, knownLastRevOrModification: 5955398 2022-07-15 04:34:38,732 [http-nio-8080-exec-10] WARN co.elastic.apm.agent.impl.transaction.Span - Max spans (500) for transaction 'RepositoryController#recoverLastRevision' 00-ed659d532656d51e9b87f5772b031c04-79c05813bbf21dae-01 (7a35462f) has been reached. For this transaction and possibly others, further spans will be dropped. See config param 'transaction_max_spans'. 2022-07-15 04:35:00.264 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 15614ms (expected ~1000ms) 2022-07-15 04:35:23.837 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 17561ms (expected ~1000ms) 2022-07-15 04:35:43.968 --- [io-8080-exec-10] : Recovery for cluster node [53]: 6106441 nodes scanned in 17.62 min (~346507/m) - last interval 143701 nodes in 65.33 s (~131987/m) 2022-07-15 04:35:43.970 --- [io-8080-exec-10] : path: /contents/documents/dci/transactions/office-transactions/17/1171100/3401500014/1401/03/17/16/register/lNCdUlTdI5cGFouzt3Sz/dci:resources, unsaved: 0, unsavedParents: 4935628, knownLastRevOrModification: 6099024 2022-07-15 04:35:43.968 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 15122ms (expected ~1000ms) 2022-07-15 04:35:58.669 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 9694ms (expected ~1000ms) 2022-07-15 04:36:10.784 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7103ms (expected ~1000ms) 2022-07-15 04:36:24.210 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 9418ms (expected ~1000ms) 2022-07-15 04:36:40.878 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 12657ms (expected ~1000ms) 2022-07-15 04:36:43.967 --- [io-8080-exec-10] : Recovery for cluster node [53]: 6293130 nodes scanned in 18.62 min (~337925/m) - last interval 186689 nodes in 60.00 s (~186689/m) 2022-07-15 04:36:43.967 --- [io-8080-exec-10] : path: /oak:index/nodetype/:index/dci%3Asubscription/contents/documents/dci/subscriptions/9/9/6/2/0/5/1/7/8/3/9962051783, unsaved: 0, unsavedParents: 5077340, knownLastRevOrModification: 6285613 2022-07-15 04:37:02.675 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 16786ms (expected ~1000ms) 2022-07-15 04:37:29.169 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 22482ms (expected ~1000ms) 2022-07-15 04:37:29.171 --- [l-1 housekeeper] : HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=45s197ms855µs235ns). 2022-07-15 04:37:41.552 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 9375ms (expected ~1000ms) 2022-07-15 04:37:43.967 --- [io-8080-exec-10] : Recovery for cluster node [53]: 6379713 nodes scanned in 19.62 min (~325116/m) - last interval 86583 nodes in 60.00 s (~86583/m) 2022-07-15 04:37:43.967 --- [io-8080-exec-10] : path: /contents/documents/dci/transactions/office-transactions/13/1131100/2501600005/1401/03/17/17/register/Yxi9tjhVSJu8FNf9cQIz/dci:resources/gam:resourceReferenceMetadata-1, unsaved: 0, unsavedParents: 5142298, knownLastRevOrModification: 6372146 2022-07-15 04:37:59.201 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 14641ms (expected ~1000ms) 2022-07-15 04:38:09.760 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7548ms (expected ~1000ms) 2022-07-15 04:38:28.695 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 15924ms (expected ~1000ms) 2022-07-15 04:38:44.301 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 12597ms (expected ~1000ms) 2022-07-15 04:38:44.301 --- [io-8080-exec-10] : Recovery for cluster node [53]: 6473584 nodes scanned in 20.63 min (~313819/m) - last interval 93871 nodes in 60.33 s (~93355/m) 2022-07-15 04:38:44.302 --- [io-8080-exec-10] : path: /oak:index/reference/:references/f64deeed-847d-47f5-b148-e89e6510cef3/contents/documents/dci, unsaved: 0, unsavedParents: 5211504, knownLastRevOrModification: 6465955 2022-07-15 04:39:01.658 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 14349ms (expected ~1000ms) 2022-07-15 04:39:19.263 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 14599ms (expected ~1000ms) 2022-07-15 04:39:30.779 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 9508ms (expected ~1000ms) 2022-07-15 04:39:42.842 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 10056ms (expected ~1000ms) 2022-07-15 04:39:42,872 [http-nio-8080-exec-10] WARN co.elastic.apm.agent.impl.transaction.Span - Max spans (500) for transaction 'RepositoryController#recoverLastRevision' 00-ed659d532656d51e9b87f5772b031c04-79c05813bbf21dae-01 (7a35462f) has been reached. For this transaction and possibly others, further spans will be dropped. See config param 'transaction_max_spans'. 2022-07-15 04:39:44.298 --- [io-8080-exec-10] : Recovery for cluster node [53]: 6581340 nodes scanned in 21.63 min (~304292/m) - last interval 107756 nodes in 60.00 s (~107756/m) 2022-07-15 04:39:44.298 --- [io-8080-exec-10] : path: /oak:index/nodetype/:index/dci%3Asubscription/contents/documents/dci/subscriptions/9/9/6/2/1/4/0/1/1/3/9962140113, unsaved: 0, unsavedParents: 5293369, knownLastRevOrModification: 6573648 2022-07-15 04:39:59.939 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 15089ms (expected ~1000ms) 2022-07-15 04:40:11.824 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 9877ms (expected ~1000ms) 2022-07-15 04:40:22.289 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8458ms (expected ~1000ms) 2022-07-15 04:40:33.651 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 9354ms (expected ~1000ms) 2022-07-15 04:40:43.950 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8290ms (expected ~1000ms) 2022-07-15 04:40:44.298 --- [io-8080-exec-10] : Recovery for cluster node [53]: 6670610 nodes scanned in 22.63 min (~294789/m) - last interval 89270 nodes in 60.00 s (~89270/m) 2022-07-15 04:40:44.298 --- [io-8080-exec-10] : path: /oak:index/reference/:references/cded8362-04e6-40e5-88b2-951be5bdd91a/contents/documents/dci/transactions, unsaved: 0, unsavedParents: 5359889, knownLastRevOrModification: 6662888 2022-07-15 04:40:54.440 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8483ms (expected ~1000ms) 2022-07-15 04:41:11.566 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 16116ms (expected ~1000ms) 2022-07-15 04:41:24.048 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 11473ms (expected ~1000ms) 2022-07-15 04:41:38.502 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 13443ms (expected ~1000ms) 2022-07-15 04:41:54.742 --- [io-8080-exec-10] : Recovery for cluster node [53]: 6746070 nodes scanned in 23.80 min (~283419/m) - last interval 75460 nodes in 70.44 s (~64273/m) 2022-07-15 04:41:54.741 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 15228ms (expected ~1000ms) 2022-07-15 04:41:54.743 --- [io-8080-exec-10] : path: /oak:index/nodetype/:index/gam%3AresourceReferenceMetadataContainer/contents/documents/dci/transactions/office-transactions/30/1301100/5301810147/1401/03/17/18/register, unsaved: 0, unsavedParents: 5416846, knownLastRevOrModification: 6738263 2022-07-15 04:42:10.369 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 14618ms (expected ~1000ms) 2022-07-15 04:42:20.855 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 9476ms (expected ~1000ms) 2022-07-15 04:42:30.426 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8562ms (expected ~1000ms) 2022-07-15 04:42:40.042 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8615ms (of which the last update operation took 7ms) (expected ~1000ms) 2022-07-15 04:42:49.795 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8751ms (expected ~1000ms) 2022-07-15 04:43:09.228 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 18427ms (expected ~1000ms) 2022-07-15 04:43:09.229 --- [io-8080-exec-10] : Recovery for cluster node [53]: 6818736 nodes scanned in 25.04 min (~272271/m) - last interval 72666 nodes in 74.49 s (~58533/m) 2022-07-15 04:43:09.230 --- [io-8080-exec-10] : path: /oak:index/reference/:references/59874920-eb52-49b4-b7fc-b3612f674c76/contents/documents/dci/transactions/office-transactions/4380/2101100/2101500111/1401/03/17/19/register/x6tPsKFYIlUMf7TGSKCR/dci:resources/gam:resourceReferenceMetadata-1/gam:reference, unsaved: 0, unsavedParents: 5470923, knownLastRevOrModification: 6810799 2022-07-15 04:43:26.217 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 15980ms (expected ~1000ms) 2022-07-15 04:43:35.639 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8416ms (expected ~1000ms) 2022-07-15 04:43:45.163 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8522ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:43:54.323 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8158ms (expected ~1000ms) 2022-07-15 04:44:04.099 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8769ms (expected ~1000ms) 2022-07-15 04:44:13.636 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8534ms (of which the last update operation took 5ms) (expected ~1000ms) 2022-07-15 04:44:13.637 --- [io-8080-exec-10] : Recovery for cluster node [53]: 6874990 nodes scanned in 26.12 min (~263234/m) - last interval 56254 nodes in 64.41 s (~52403/m) 2022-07-15 04:44:13.637 --- [io-8080-exec-10] : path: /oak:index/reference/:references/a96bb1e9-bb89-4713-ba8e-58c9ce072fc0/contents/documents/dci/transactions/office-transactions/5/2151100/7101500053/1401/03, unsaved: 0, unsavedParents: 5513379, knownLastRevOrModification: 6866923 2022-07-15 04:44:28.944 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 15306ms (of which the last update operation took 2ms) (expected ~1000ms) 2022-07-15 04:44:38.792 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 9850ms (of which the last update operation took 9ms) (expected ~1000ms) 2022-07-15 04:44:48.224 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 9432ms (of which the last update operation took 2ms) (expected ~1000ms) 2022-07-15 04:44:48,265 [http-nio-8080-exec-10] WARN co.elastic.apm.agent.impl.transaction.Span - Max spans (500) for transaction 'RepositoryController#recoverLastRevision' 00-ed659d532656d51e9b87f5772b031c04-79c05813bbf21dae-01 (7a35462f) has been reached. For this transaction and possibly others, further spans will be dropped. See config param 'transaction_max_spans'. 2022-07-15 04:44:57.450 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 9227ms (of which the last update operation took 8ms) (expected ~1000ms) 2022-07-15 04:45:08.319 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 10869ms (of which the last update operation took 1ms) (expected ~1000ms) 2022-07-15 04:45:17.195 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8876ms (of which the last update operation took 11ms) (expected ~1000ms) 2022-07-15 04:45:17.198 --- [io-8080-exec-10] : Recovery for cluster node [53]: 6918332 nodes scanned in 27.18 min (~254568/m) - last interval 43342 nodes in 63.56 s (~40913/m) 2022-07-15 04:45:17.198 --- [io-8080-exec-10] : path: /oak:index/reference/:references/5937e8d3-7106-4c65-b449-35ce61561449/contents/documents/dci/transactions/office-transactions/31/1311137/2209500003/1401/03/17/19/register/pmHPiPcZTou5FWtWsBUm/dci:resources/gam:resourceReferenceMetadata-1, unsaved: 0, unsavedParents: 5545456, knownLastRevOrModification: 6910155 2022-07-15 04:45:29.904 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 12706ms (of which the last update operation took 2ms) (expected ~1000ms) 2022-07-15 04:45:43.182 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 13281ms (of which the last update operation took 10ms) (expected ~1000ms) 2022-07-15 04:45:51.416 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8234ms (of which the last update operation took 9ms) (expected ~1000ms) 2022-07-15 04:46:00.312 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8895ms (of which the last update operation took 1ms) (expected ~1000ms) 2022-07-15 04:46:16.446 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 16134ms (of which the last update operation took 7ms) (expected ~1000ms) 2022-07-15 04:46:34.829 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 18384ms (of which the last update operation took 8ms) (expected ~1000ms) 2022-07-15 04:46:34.835 --- [io-8080-exec-10] : Recovery for cluster node [53]: 6951277 nodes scanned in 28.47 min (~244156/m) - last interval 32945 nodes in 77.64 s (~25461/m) 2022-07-15 04:46:34.835 --- [io-8080-exec-10] : path: /oak:index/nodetype/:index/gam%3Afolder/contents/documents/dci/transactions/office-transactions/2/2121111/4208813280/1401/03/17/20, unsaved: 0, unsavedParents: 5569659, knownLastRevOrModification: 6943020 2022-07-15 04:46:50.619 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 15785ms (of which the last update operation took 13ms) (expected ~1000ms) 2022-07-15 04:47:10.497 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 19881ms (of which the last update operation took 16ms) (expected ~1000ms) 2022-07-15 04:47:26.476 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 15977ms (of which the last update operation took 21ms) (expected ~1000ms) 2022-07-15 04:47:41.611 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 15136ms (of which the last update operation took 16ms) (expected ~1000ms) 2022-07-15 04:47:41.613 --- [io-8080-exec-10] : Recovery for cluster node [53]: 6964227 nodes scanned in 29.58 min (~235409/m) - last interval 12950 nodes in 66.77 s (~11636/m) 2022-07-15 04:47:41.615 --- [io-8080-exec-10] : path: /oak:index/reference/:references/03cd8a24-2395-4ea4-920d-22d341b2aaa5/contents/documents, unsaved: 0, unsavedParents: 5579320, knownLastRevOrModification: 6955930 2022-07-15 04:47:50.028 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8420ms (of which the last update operation took 10ms) (expected ~1000ms) 2022-07-15 04:47:59.167 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 9139ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:48:07.511 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8344ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:48:16.238 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8725ms (of which the last update operation took 4ms) (expected ~1000ms) 2022-07-15 04:48:30.797 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 14561ms (of which the last update operation took 9ms) (expected ~1000ms) 2022-07-15 04:48:39.047 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8247ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:48:47.730 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8686ms (of which the last update operation took 3ms) (expected ~1000ms) 2022-07-15 04:48:47.731 --- [io-8080-exec-10] : Recovery for cluster node [53]: 6987960 nodes scanned in 30.69 min (~227728/m) - last interval 23733 nodes in 66.12 s (~21535/m) 2022-07-15 04:48:47.732 --- [io-8080-exec-10] : path: /oak:index/reference/:references/7e73d359-cb29-40b4-8b44-c564241e7d21/contents/documents/dci/transactions/office-transactions/31/1311100/2201500085/1401/03/17/20/register, unsaved: 0, unsavedParents: 5596679, knownLastRevOrModification: 6979593 2022-07-15 04:48:55.515 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7784ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:49:03.715 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8200ms (of which the last update operation took 2ms) (expected ~1000ms) 2022-07-15 04:49:11.424 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7710ms (of which the last update operation took 7ms) (expected ~1000ms) 2022-07-15 04:49:18.677 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7252ms (of which the last update operation took 1ms) (expected ~1000ms) 2022-07-15 04:49:26.551 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7875ms (of which the last update operation took 11ms) (expected ~1000ms) 2022-07-15 04:49:35.157 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8606ms (of which the last update operation took 1ms) (expected ~1000ms) 2022-07-15 04:49:43.159 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7999ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:49:50.605 --- [io-8080-exec-10] : Recovery for cluster node [53]: 7009255 nodes scanned in 31.73 min (~220878/m) - last interval 21295 nodes in 62.88 s (~20321/m) 2022-07-15 04:49:50.605 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7448ms (of which the last update operation took 4ms) (expected ~1000ms) 2022-07-15 04:49:50.606 --- [io-8080-exec-10] : path: /oak:index/uuid/:index/c38cd51b-7d4e-418a-896d-1fd171a097f3, unsaved: 0, unsavedParents: 5612470, knownLastRevOrModification: 7000828 2022-07-15 04:49:50,612 [http-nio-8080-exec-10] WARN co.elastic.apm.agent.impl.transaction.Span - Max spans (500) for transaction 'RepositoryController#recoverLastRevision' 00-ed659d532656d51e9b87f5772b031c04-79c05813bbf21dae-01 (7a35462f) has been reached. For this transaction and possibly others, further spans will be dropped. See config param 'transaction_max_spans'. 2022-07-15 04:49:57.974 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7370ms (of which the last update operation took 8ms) (expected ~1000ms) 2022-07-15 04:50:04.999 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7025ms (of which the last update operation took 4ms) (expected ~1000ms) 2022-07-15 04:50:12.275 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7275ms (of which the last update operation took 7ms) (expected ~1000ms) 2022-07-15 04:50:19.766 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7492ms (of which the last update operation took 2ms) (expected ~1000ms) 2022-07-15 04:50:26.509 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6743ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:50:33.701 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7191ms (of which the last update operation took 1ms) (expected ~1000ms) 2022-07-15 04:50:41.692 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7992ms (of which the last update operation took 9ms) (expected ~1000ms) 2022-07-15 04:50:48.859 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7167ms (of which the last update operation took 3ms) (expected ~1000ms) 2022-07-15 04:50:56.372 --- [io-8080-exec-10] : Recovery for cluster node [53]: 7025490 nodes scanned in 32.83 min (~213998/m) - last interval 16235 nodes in 65.77 s (~14811/m) 2022-07-15 04:50:56.371 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7511ms (of which the last update operation took 5ms) (expected ~1000ms) 2022-07-15 04:50:56.372 --- [io-8080-exec-10] : path: /oak:index/reference/:references/aaec2da3-6f34-4f87-847a-e27b0feeec9e/contents/documents/dci/transactions/office-transactions/4380/2101100/2101500283/1401/03, unsaved: 0, unsavedParents: 5624388, knownLastRevOrModification: 7017033 2022-07-15 04:51:02.798 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6427ms (of which the last update operation took 2ms) (expected ~1000ms) 2022-07-15 04:51:09.548 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6750ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:51:17.258 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7711ms (of which the last update operation took 2ms) (expected ~1000ms) 2022-07-15 04:51:24.892 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7633ms (of which the last update operation took 5ms) (expected ~1000ms) 2022-07-15 04:51:36.486 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 11594ms (of which the last update operation took 2ms) (expected ~1000ms) 2022-07-15 04:51:43.778 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7292ms (of which the last update operation took 7ms) (expected ~1000ms) 2022-07-15 04:51:51.843 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 8066ms (of which the last update operation took 1ms) (expected ~1000ms) 2022-07-15 04:51:58.501 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6658ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:51:58.502 --- [io-8080-exec-10] : Recovery for cluster node [53]: 7035066 nodes scanned in 33.87 min (~207737/m) - last interval 9576 nodes in 62.13 s (~9247/m) 2022-07-15 04:51:58.503 --- [io-8080-exec-10] : path: /oak:index/nodetype/:index/dci%3Asubscription/contents/documents/dci/subscriptions/9/9/1/5/9/5/4/0/9/0, unsaved: 0, unsavedParents: 5631344, knownLastRevOrModification: 7026584 2022-07-15 04:52:06.002 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7500ms (of which the last update operation took 1ms) (expected ~1000ms) 2022-07-15 04:52:12.629 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6627ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:52:27.145 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 14516ms (of which the last update operation took 1ms) (expected ~1000ms) 2022-07-15 04:52:41.294 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 14149ms (of which the last update operation took 10ms) (expected ~1000ms) 2022-07-15 04:52:56.336 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 15042ms (of which the last update operation took 7ms) (expected ~1000ms) 2022-07-15 04:53:03.179 --- [io-8080-exec-10] : Recovery for cluster node [53]: 7039909 nodes scanned in 34.94 min (~201468/m) - last interval 4843 nodes in 64.68 s (~4492/m) 2022-07-15 04:53:03.179 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6843ms (of which the last update operation took 8ms) (expected ~1000ms) 2022-07-15 04:53:03.180 --- [io-8080-exec-10] : path: /oak:index/uuid/:index/450dcdb6-1336-467c-8850-5b39af63906b, unsaved: 0, unsavedParents: 5634851, knownLastRevOrModification: 7031417 2022-07-15 04:53:11.160 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7981ms (of which the last update operation took 2ms) (expected ~1000ms) 2022-07-15 04:53:17.223 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6063ms (of which the last update operation took 7ms) (expected ~1000ms) 2022-07-15 04:53:25.107 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7885ms (of which the last update operation took 1ms) (expected ~1000ms) 2022-07-15 04:53:32.416 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7308ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:53:39.797 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7382ms (of which the last update operation took 5ms) (expected ~1000ms) 2022-07-15 04:53:46.050 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6251ms (of which the last update operation took 7ms) (expected ~1000ms) 2022-07-15 04:53:52.042 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 5994ms (of which the last update operation took 2ms) (expected ~1000ms) 2022-07-15 04:53:58.354 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6311ms (of which the last update operation took 5ms) (expected ~1000ms) 2022-07-15 04:54:04.229 --- [io-8080-exec-10] : Recovery for cluster node [53]: 7045593 nodes scanned in 35.96 min (~195925/m) - last interval 5684 nodes in 61.05 s (~5586/m) 2022-07-15 04:54:04.229 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 5876ms (of which the last update operation took 2ms) (expected ~1000ms) 2022-07-15 04:54:04.230 --- [io-8080-exec-10] : path: /oak:index/reference/:references/65e4c84b-da85-4aa6-b887-6095afda47cb/contents/documents/dci/transactions/office-transactions/28, unsaved: 0, unsavedParents: 5638872, knownLastRevOrModification: 7037086 2022-07-15 04:54:11.418 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7189ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:54:17.599 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6181ms (of which the last update operation took 2ms) (expected ~1000ms) 2022-07-15 04:54:23.713 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6114ms (of which the last update operation took 5ms) (expected ~1000ms) 2022-07-15 04:54:30.062 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6348ms (of which the last update operation took 2ms) (expected ~1000ms) 2022-07-15 04:54:36.455 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6394ms (of which the last update operation took 9ms) (expected ~1000ms) 2022-07-15 04:54:42.490 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6035ms (expected ~1000ms) 2022-07-15 04:54:48.719 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6228ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:54:54.604 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 5886ms (of which the last update operation took 3ms) (expected ~1000ms) 2022-07-15 04:55:00.513 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 5908ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:55:00,551 [http-nio-8080-exec-10] WARN co.elastic.apm.agent.impl.transaction.Span - Max spans (500) for transaction 'RepositoryController#recoverLastRevision' 00-ed659d532656d51e9b87f5772b031c04-79c05813bbf21dae-01 (7a35462f) has been reached. For this transaction and possibly others, further spans will be dropped. See config param 'transaction_max_spans'. 2022-07-15 04:55:07.775 --- [io-8080-exec-10] : Recovery for cluster node [53]: 7049786 nodes scanned in 37.02 min (~190433/m) - last interval 4193 nodes in 63.55 s (~3959/m) 2022-07-15 04:55:07.775 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 7263ms (of which the last update operation took 1ms) (expected ~1000ms) 2022-07-15 04:55:07.778 --- [io-8080-exec-10] : path: /contents/documents/dci/subscriptions/9/9/6/2/3/5/5/1/4/9, unsaved: 0, unsavedParents: 5641978, knownLastRevOrModification: 7041269 2022-07-15 04:55:13.997 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6221ms (of which the last update operation took 8ms) (expected ~1000ms) 2022-07-15 04:55:20.284 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6288ms (of which the last update operation took 1ms) (expected ~1000ms) 2022-07-15 04:55:26.806 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6521ms (of which the last update operation took 5ms) (expected ~1000ms) 2022-07-15 04:55:32.810 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6005ms (of which the last update operation took 1ms) (expected ~1000ms) 2022-07-15 04:55:39.000 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6189ms (of which the last update operation took 7ms) (expected ~1000ms) 2022-07-15 04:55:45.340 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6341ms (of which the last update operation took 1ms) (expected ~1000ms) 2022-07-15 04:55:51.544 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6204ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:55:51.553 --- [ate thread (56)] : Background operations stats (clean:0, split:0, refresh:2, sweep:0, lock:0, write:12551, num:1, calls:1, totalWriteTime:12554) 2022-07-15 04:55:57.791 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6247ms (of which the last update operation took 1ms) (expected ~1000ms) 2022-07-15 04:56:04.008 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6216ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:56:10.545 --- [io-8080-exec-10] : Recovery for cluster node [53]: 7052045 nodes scanned in 38.07 min (~185259/m) - last interval 2259 nodes in 62.77 s (~2159/m) 2022-07-15 04:56:10.546 --- [io-8080-exec-10] : path: /contents/documents/dci/transactions/office-transactions/12/1121100/2301500020/1401/03/17/20/transfer/GrC6HbfYIgc8TxUmtyS7, unsaved: 0, unsavedParents: 5643680, knownLastRevOrModification: 7043528 2022-07-15 04:56:10.545 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6538ms (of which the last update operation took 3ms) (expected ~1000ms) 2022-07-15 04:56:16.513 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 5968ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:56:22.627 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6114ms (of which the last update operation took 1ms) (expected ~1000ms) 2022-07-15 04:56:28.662 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6034ms (of which the last update operation took 7ms) (expected ~1000ms) 2022-07-15 04:56:34.742 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6081ms (of which the last update operation took 1ms) (expected ~1000ms) 2022-07-15 04:56:40.680 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 5937ms (of which the last update operation took 7ms) (expected ~1000ms) 2022-07-15 04:56:46.680 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6001ms (of which the last update operation took 2ms) (expected ~1000ms) 2022-07-15 04:56:53.002 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6321ms (of which the last update operation took 5ms) (expected ~1000ms) 2022-07-15 04:56:59.015 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6014ms (of which the last update operation took 1ms) (expected ~1000ms) 2022-07-15 04:57:05.068 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6053ms (of which the last update operation took 9ms) (expected ~1000ms) 2022-07-15 04:57:11.532 --- [io-8080-exec-10] : Recovery for cluster node [53]: 7053867 nodes scanned in 39.08 min (~180487/m) - last interval 1822 nodes in 60.99 s (~1792/m) 2022-07-15 04:57:11.532 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6464ms (of which the last update operation took 1ms) (expected ~1000ms) 2022-07-15 04:57:11.533 --- [io-8080-exec-10] : path: /oak:index/nodetype/:index/dci%3Acustomer/contents/documents/dci/customers/individuals/3/3/6/9/3/5/0/5, unsaved: 0, unsavedParents: 5644978, knownLastRevOrModification: 7045345 2022-07-15 04:57:17.550 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6018ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:57:24.496 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6945ms (of which the last update operation took 2ms) (expected ~1000ms) 2022-07-15 04:57:30.516 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6019ms (of which the last update operation took 10ms) (expected ~1000ms) 2022-07-15 04:57:36.524 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 6010ms (of which the last update operation took 2ms) (expected ~1000ms) java.lang.OutOfMemoryError: GC overhead limit exceeded Dumping heap to "/gam/temp" ... Unable to create "/gam/temp": No such file or directory 2022-07-15 04:57:51.050 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 14525ms (of which the last update operation took 6ms) (expected ~1000ms) 2022-07-15 04:58:19.678 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 13896ms (of which the last update operation took 9ms) (expected ~1000ms) 2022-07-15 04:58:20.179 --- [io-8080-exec-10] : Released recovery lock for cluster id 53 (recovery successful: false) 2022-07-15 04:58:20.186 --- [ead thread (56)] : Background read operations stats (read:15241 ReadStats{cacheStats:NOP, head:15239, cache:0, diff: 0, lock:0, dispatch:0, numExternalChanges:0, externalChangesLag:0, totalReadTime:15241}) 2022-07-15 04:58:20.233 --- [io-8080-exec-10] : GSR-000 is happened with message : Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: GC overhead limit exceeded org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.OutOfMemoryError: GC overhead limit exceeded at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1079) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) at javax.servlet.http.HttpServlet.service(HttpServlet.java:681) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:764) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:769) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1726) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded at oracle.jdbc.driver.BlockSource$ThreadedCachingBlockSource.get(BlockSource.java:522) at oracle.jdbc.driver.DynamicByteArray.put(DynamicByteArray.java:502) at oracle.jdbc.driver.ByteArray.put(ByteArray.java:216) at oracle.jdbc.driver.StringBinder.bind(OraclePreparedStatement.java:18836) at oracle.jdbc.driver.OraclePreparedStatement.setupBindBuffers(OraclePreparedStatement.java:3243) at oracle.jdbc.driver.OraclePreparedStatement.processCompletedBindRow(OraclePreparedStatement.java:2661) at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3772) at oracle.jdbc.driver.T4CPreparedStatement.executeInternal(T4CPreparedStatement.java:1343) at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3822) at oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery(OraclePreparedStatementWrapper.java:1165) at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java) at org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStoreJDBC.read(RDBDocumentStoreJDBC.java:867) at org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStore.readDocumentUncached(RDBDocumentStore.java:1957) at org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStore.access$300(RDBDocumentStore.java:251) at org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStore$2.call(RDBDocumentStore.java:1540) at org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStore$2.call(RDBDocumentStore.java:1537) at org.apache.jackrabbit.oak.plugins.document.cache.NodeDocumentCache$1.call(NodeDocumentCache.java:179) at org.apache.jackrabbit.oak.plugins.document.cache.NodeDocumentCache$1.call(NodeDocumentCache.java:173) at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.load(CacheLIRS.java:1027) at org.apache.jackrabbit.oak.cache.CacheLIRS$Segment.get(CacheLIRS.java:988) at org.apache.jackrabbit.oak.cache.CacheLIRS.get(CacheLIRS.java:299) at org.apache.jackrabbit.oak.plugins.document.cache.NodeDocumentCache.get(NodeDocumentCache.java:185) at org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStore.readDocumentCached(RDBDocumentStore.java:1537) at org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStore.find(RDBDocumentStore.java:281) at org.apache.jackrabbit.oak.plugins.document.rdb.RDBDocumentStore.find(RDBDocumentStore.java:276) at org.apache.jackrabbit.oak.plugins.document.NodeDocument.getPreviousDocument(NodeDocument.java:1306) at org.apache.jackrabbit.oak.plugins.document.NodeDocument.getPreviousDocs(NodeDocument.java:1275) at org.apache.jackrabbit.oak.plugins.document.ValueMap$2.containsKey(ValueMap.java:169) at org.apache.jackrabbit.oak.plugins.document.NodeDocument.getPreviousDocs(NodeDocument.java:1277) at org.apache.jackrabbit.oak.plugins.document.ValueMap$2.containsKey(ValueMap.java:169) at org.apache.jackrabbit.oak.plugins.document.NodeDocument.getPreviousDocs(NodeDocument.java:1277) 2022-07-15 04:58:21.176 --- [ate thread (56)] : BackgroundLeaseUpdate.execute: time since last renewClusterIdLease() call longer than expected: 16231ms (of which the last update operation took 15231ms) (expected ~1000ms)