[SLING-10743] Multiple instances cleaning up distribution queue at the same time and causing commit failures

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: Content Distribution Core 0.4.8
• Component/s: Content Distribution
• Assignee: Timothee Maret
• Reporter: Alexei Krainiouk
• Created: 18/Aug/21 21:36
• Updated: 21/Dec/21 12:58
• Resolved: 19/Aug/21 10:38

Description

There doesn't seem to be an arbitration between different server instances when distribution queue cleanup is concerned. This occasionally causes commit failures inside org.apache.sling.distribution.packaging.impl.ResourceDistributionPackageCleanup class with the folllowing exception thrown:

org.apache.sling.distribution.packaging.impl.ResourceDistributionPackageCleanup Failed to delete disposable packages
org.apache.sling.api.resource.PersistenceException: Unable to commit changes to session.
        at org.apache.sling.jcr.resource.internal.helper.jcr.JcrResourceProvider.commit(JcrResourceProvider.java:516) [org.apache.sling.jcr.resource:3.0.22]
        at org.apache.sling.resourceresolver.impl.providers.stateful.AuthenticatedResourceProvider.commit(AuthenticatedResourceProvider.java:215) [org.apache.sling.resourceresolver:1.7.4]
        at org.apache.sling.resourceresolver.impl.helper.ResourceResolverControl.commit(ResourceResolverControl.java:425) [org.apache.sling.resourceresolver:1.7.4]
        at org.apache.sling.resourceresolver.impl.ResourceResolverImpl.commit(ResourceResolverImpl.java:1005) [org.apache.sling.resourceresolver:1.7.4]
        at org.apache.sling.distribution.packaging.impl.ResourceDistributionPackageCleanup.run(ResourceDistributionPackageCleanup.java:71) [org.apache.sling.distribution.core:0.4.5.T202105062218-732
6f21]
        at org.apache.sling.commons.scheduler.impl.QuartzJobExecutor.execute(QuartzJobExecutor.java:349) [org.apache.sling.commons.scheduler:2.7.12]
        at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [org.apache.sling.commons.scheduler:2.7.12]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: javax.jcr.InvalidItemStateException: OakState0001: Unresolved conflicts in /var/sling/distribution/packages/signed-url/data
        at org.apache.jackrabbit.oak.api.CommitFailedException.asRepositoryException(CommitFailedException.java:238) [org.apache.jackrabbit.oak-api:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.api.CommitFailedException.asRepositoryException(CommitFailedException.java:213) [org.apache.jackrabbit.oak-api:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.newRepositoryException(SessionDelegate.java:684) [org.apache.jackrabbit.oak-jcr:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.save(SessionDelegate.java:505) [org.apache.jackrabbit.oak-jcr:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.jcr.session.SessionImpl$8.performVoid(SessionImpl.java:429) [org.apache.jackrabbit.oak-jcr:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.jcr.delegate.SessionDelegate.performVoid(SessionDelegate.java:275) [org.apache.jackrabbit.oak-jcr:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.jcr.session.SessionImpl.save(SessionImpl.java:426) [org.apache.jackrabbit.oak-jcr:1.39.0.R1889746]
        at com.adobe.granite.repository.impl.CRX3SessionImpl.save(CRX3SessionImpl.java:207) [com.adobe.granite.repository:1.6.128]
        at org.apache.sling.jcr.resource.internal.helper.jcr.JcrResourceProvider.commit(JcrResourceProvider.java:514) [org.apache.sling.jcr.resource:3.0.22]
        ... 9 common frames omitted
Caused by: org.apache.jackrabbit.oak.api.CommitFailedException: OakState0001: Unresolved conflicts in /var/sling/distribution/packages/signed-url/data
        at org.apache.jackrabbit.oak.plugins.commit.ConflictValidator.failOnMergeConflict(ConflictValidator.java:115) [org.apache.jackrabbit.oak-core:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.plugins.commit.ConflictValidator.propertyAdded(ConflictValidator.java:84) [org.apache.jackrabbit.oak-core:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.spi.commit.CompositeEditor.propertyAdded(CompositeEditor.java:82) [org.apache.jackrabbit.oak-store-spi:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.spi.commit.EditorDiff.propertyAdded(EditorDiff.java:81) [org.apache.jackrabbit.oak-store-spi:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:394) [org.apache.jackrabbit.oak-store-spi:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:147) [org.apache.jackrabbit.oak-store-spi:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:416) [org.apache.jackrabbit.oak-store-spi:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:147) [org.apache.jackrabbit.oak-store-spi:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:416) [org.apache.jackrabbit.oak-store-spi:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:147) [org.apache.jackrabbit.oak-store-spi:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:416) [org.apache.jackrabbit.oak-store-spi:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:147) [org.apache.jackrabbit.oak-store-spi:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:416) [org.apache.jackrabbit.oak-store-spi:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:147) [org.apache.jackrabbit.oak-store-spi:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:416) [org.apache.jackrabbit.oak-store-spi:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.spi.commit.EditorDiff.childNodeChanged(EditorDiff.java:147) [org.apache.jackrabbit.oak-store-spi:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.composite.CompositeNodeState$WrappingDiff.childNodeChanged(CompositeNodeState.java:309) [org.apache.jackrabbit.oak-store-composite:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.composite.CompositeNodeState$ChildrenDiffFilter.childNodeChanged(CompositeNodeState.java:256) [org.apache.jackrabbit.oak-store-composite:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.plugins.memory.ModifiedNodeState.compareAgainstBaseState(ModifiedNodeState.java:416) [org.apache.jackrabbit.oak-store-spi:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.plugins.document.ModifiedDocumentNodeState.compareAgainstBaseState(ModifiedDocumentNodeState.java:132) [org.apache.jackrabbit.oak-store-document:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.composite.CompositeNodeState.compareAgainstBaseState(CompositeNodeState.java:163) [org.apache.jackrabbit.oak-store-composite:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.spi.commit.EditorDiff.process(EditorDiff.java:51) [org.apache.jackrabbit.oak-store-spi:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.spi.commit.EditorHook.processCommit(EditorHook.java:54) [org.apache.jackrabbit.oak-store-spi:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.spi.commit.CompositeHook.processCommit(CompositeHook.java:60) [org.apache.jackrabbit.oak-store-spi:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.spi.commit.CompositeHook.processCommit(CompositeHook.java:60) [org.apache.jackrabbit.oak-store-spi:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.composite.CommitHookEnhancer.processCommit(CommitHookEnhancer.java:55) [org.apache.jackrabbit.oak-store-composite:1.39.0.R1889746]
        at org.apache.jackrabbit.oak.plugins.document.TimingHook.processCommit(TimingHook.java:59) [org.apache.jackrabbit.oak-store-document:1.39.0.R1889746]