[OAK-9115] Command failed with error 17419: 'Resulting document after update is larger than 16777216'

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.8.22
• Component/s: mongomk

Environment

Oak: 1.8.22
Mongo Node Store
S3 Blob Store

• Assignee: Unassigned
• Reporter: David Snowsill
• Created: 15/Jun/20 19:16
• Updated: 29/Jun/20 21:55
• Resolved: 29/Jun/20 12:38

Description

When attempting to save a new item of content into Oak, we are now getting the following message. As mentioned in a previous issue raised (OAK-9112), we have recently backloaded a lot of content into Oak with the Mongo "nodes" collection increasing from 100,000 documents to 600,000 documents.

The new content we are creating is less than 1k in size.

I appreciate this probably not a bug but any guidance resolving this issue in a production system would be appreciated.

We can update existing content items without issue only creating new items is causing the stack trace below.

16 Jun 2020 08:03:34.263HostName=crms-service-content-f899d5f6d-lwmgp 08:03:34.223 [content-service] [http-nio-8080-exec-195] ERROR o.s.c.s.instrument.web.TraceFilter - Uncaught exception thrown
org.springframework.web.util.NestedServletException: Request processing failed; nested exception is javax.jcr.InvalidItemStateException: OakMerge0001: OakMerge0001: Failed to merge changes to the underlying store (retries 5, 70255 ms)

15 Jun 2020 19:07:33.228HostName=crms-service-content-f899d5f6d-qr9lt 19:07:33.126 [content-service] [http-nio-8080-exec-311] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is javax.jcr.InvalidItemStateException: OakMerge0001: OakMerge0001: Failed to merge changes to the underlying store (retries 5, 68221 ms)] with root cause
com.mongodb.MongoCommandException: Command failed with error 17419: 'Resulting document after update is larger than 16777216' on server sg-pbprod-10448.servers.mongodirector.com:27017. The full response is { "ok" : 0.0, "errmsg" : "Resulting document after update is larger than 16777216", "code" : 17419, "codeName" : "Location17419" }
	at com.mongodb.connection.ProtocolHelper.getCommandFailureException(ProtocolHelper.java:115)
	at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:114)
	at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:168)
	at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:289)
	at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:176)
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:216)
	at com.mongodb.operation.CommandOperationHelper.executeWrappedCommandProtocol(CommandOperationHelper.java:153)
	at com.mongodb.operation.FindAndUpdateOperation$1.call(FindAndUpdateOperation.java:335)
	at com.mongodb.operation.OperationHelper.withConnectionSource(OperationHelper.java:426)
	at com.mongodb.operation.OperationHelper.withConnection(OperationHelper.java:417)
	at com.mongodb.operation.FindAndUpdateOperation.execute(FindAndUpdateOperation.java:331)
	at com.mongodb.Mongo.execute(Mongo.java:845)
	at com.mongodb.Mongo$2.execute(Mongo.java:828)
	at com.mongodb.DBCollection.findAndModify(DBCollection.java:1928)
	at com.mongodb.DBCollection.findAndModify(DBCollection.java:1792)
	at com.mongodb.DBCollection.findAndModify(DBCollection.java:1761)
	at com.mongodb.DBCollection.findAndModify(DBCollection.java:1708)
	at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.findAndModify(MongoDocumentStore.java:862)
	at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.createOrUpdate(MongoDocumentStore.java:908)
	at org.apache.jackrabbit.oak.plugins.document.mongo.MongoDocumentStore.createOrUpdate(MongoDocumentStore.java:987)
	at org.apache.jackrabbit.oak.plugins.document.util.LeaseCheckDocumentStoreWrapper.createOrUpdate(LeaseCheckDocumentStoreWrapper.java:136)
	at org.apache.jackrabbit.oak.plugins.document.Commit.applyToDocumentStore(Commit.java:343)
	at org.apache.jackrabbit.oak.plugins.document.Commit.applyToDocumentStore(Commit.java:264)
	at org.apache.jackrabbit.oak.plugins.document.Commit.applyInternal(Commit.java:233)
	at org.apache.jackrabbit.oak.plugins.document.Commit.apply(Commit.java:222)
	at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreBranch.persist(DocumentNodeStoreBranch.java:301)
	at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreBranch.persist(DocumentNodeStoreBranch.java:271)
	at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreBranch.access$400(DocumentNodeStoreBranch.java:57)
	at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreBranch$InMemory.merge(DocumentNodeStoreBranch.java:519)
	at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreBranch.merge0(DocumentNodeStoreBranch.java:188)
	at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStoreBranch.merge(DocumentNodeStoreBranch.java:130)
	at org.apache.jackrabbit.oak.plugins.document.DocumentRootBuilder.merge(DocumentRootBuilder.java:163)
	at org.apache.jackrabbit.oak.plugins.document.DocumentNodeStore.merge(DocumentNodeStore.java:1914)
	at org.apache.jackrabbit.oak.core.MutableRoot.commit(MutableRoot.java:250)
	at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.commit(SessionDelegate.java:346)
	at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.save(SessionDelegate.java:493)
	at org.apache.jackrabbit.oak.jcr.session.SessionImpl$8.performVoid(SessionImpl.java:424)
	at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.performVoid(SessionDelegate.java:273)
	at org.apache.jackrabbit.oak.jcr.session.SessionImpl.save(SessionImpl.java:421)
	at au.com.calibreft.service.content.dao.JcrRepository.executeWithSession(JcrRepository.java:527)
	at au.com.calibreft.service.content.dao.JcrRepository.executeSaveWithSession(JcrRepository.java:494)
	at au.com.calibreft.service.content.dao.JcrRepository.save(JcrRepository.java:472)
	at au.com.calibreft.service.content.dao.JcrRepository$$FastClassBySpringCGLIB$$b81905ec.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:736)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:671)
	at au.com.calibreft.service.content.dao.JcrRepository$$EnhancerBySpringCGLIB$$a5620910.save(<generated>)
	at au.com.calibreft.service.content.rest.ContentController.saveContent(ContentController.java:126)
	at sun.reflect.GeneratedMethodAccessor286.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:854)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:765)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
	at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:660)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:54)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)