[UNOMI-784] Timeout on updateByQuery request such as scoring update

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: unomi-2.3.0, unomi-1.9.0
• Fix Version/s: unomi-2.4.0, unomi-1.9.1
• Component/s: unomi(-core)
• Assignee: David Griffon
• Reporter: Serge Huber
• Created: 23/Jun/23 14:14
• Updated: 09/Aug/23 13:45
• Resolved: 08/Aug/23 15:02

Description

As we have refactoring updates that were previously done with scroll queries with updateByQueries, on systems with large data set we are now reaching timeout while waiting for the updates to complete.

Here is an example of such an error:

Error while executing in class loader 
java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-296 [ACTIVE]
	at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:773) ~[!/:?]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218) ~[!/:?]
	at org.elasticsearch.client.RestClient.performRequest(RestClient.java:205) ~[!/:?]
	at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1454) ~[!/:?]
	at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1424) ~[!/:?]
	at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1394) ~[!/:?]
	at org.elasticsearch.client.RestHighLevelClient.updateByQuery(RestHighLevelClient.java:554) ~[!/:?]
	at org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$10.execute(ElasticSearchPersistenceServiceImpl.java:1023) ~[!/:?]
	at org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$10.execute(ElasticSearchPersistenceServiceImpl.java:1005) ~[!/:?]
	at org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.executeInClassLoader(ElasticSearchPersistenceServiceImpl.java:2267) [!/:?]
	at org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl$InClassLoaderExecute.catchingExecuteInClassLoader(ElasticSearchPersistenceServiceImpl.java:2278) [!/:?]
	at org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.updateWithQueryAndScript(ElasticSearchPersistenceServiceImpl.java:1050) [!/:?]
	at org.apache.unomi.persistence.elasticsearch.ElasticSearchPersistenceServiceImpl.updateWithQueryAndStoredScript(ElasticSearchPersistenceServiceImpl.java:1001) [!/:?]
	at Proxy952471af_195a_464e_aca9_365c7d2c5bac.updateWithQueryAndStoredScript(Unknown Source) [?:?]
	at org.apache.unomi.services.impl.segments.SegmentServiceImpl.updateExistingProfilesForScoring(SegmentServiceImpl.java:1192) [!/:?]
	at org.apache.unomi.services.impl.segments.SegmentServiceImpl.recalculatePastEventConditions(SegmentServiceImpl.java:982) [!/:?]
	at org.apache.unomi.services.impl.segments.SegmentServiceImpl$1.run(SegmentServiceImpl.java:1215) [!/:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.net.SocketTimeoutException: 30,000 milliseconds timeout on connection http-outgoing-296 [ACTIVE]
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:387) ~[?:?]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92) ~[?:?]
	at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39) ~[?:?]
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175) ~[?:?]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263) ~[?:?]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492) ~[?:?]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213) ~[?:?]
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[?:?]
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[?:?]
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591) ~[?:?]