15:45:40.697 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1105233665092179142\junit9185588256459436236, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:45:40.697 INFO [main] FileStore.java:240 TarMK opened: target\junit1105233665092179142\junit9185588256459436236 (mmap=false) 15:45:40.697 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1105233665092179142\junit1365851795743929894, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:45:40.713 INFO [main] FileStore.java:240 TarMK opened: target\junit1105233665092179142\junit1365851795743929894 (mmap=false) 15:45:40.847 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:45:40.848 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:45:41.020 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x3867f159]' 15:45:41.020 DEBUG [main] StandbyServer.java:249 Binding was successful 15:45:41.274 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 8a7b2823-22fe-4fd2-b975-265592eeca93 for current head 15:45:41.274 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:52748 is allowed 15:45:41.274 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:45:41.274 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.274 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.274 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 9e6fec1d-62b6-49e1-afb1-a608ca87405f.0000000c to client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.274 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:45:41.274 DEBUG [standby-run-23] StandbyClientSyncExecution.java:103 Found missing segment 9e6fec1d-62b6-49e1-afb1-a608ca87405f 15:45:41.274 DEBUG [standby-run-23] StandbyClientSyncExecution.java:124 Inspecting segment 9e6fec1d-62b6-49e1-afb1-a608ca87405f 15:45:41.274 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 8a7b2823-22fe-4fd2-b975-265592eeca93 for references of segment 9e6fec1d-62b6-49e1-afb1-a608ca87405f 15:45:41.274 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:45:41.274 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 9e6fec1d-62b6-49e1-afb1-a608ca87405f for client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.274 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 9e6fec1d-62b6-49e1-afb1-a608ca87405f to client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.274 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:45:41.274 DEBUG [standby-run-23] StandbyClientSyncExecution.java:184 Found reference from 9e6fec1d-62b6-49e1-afb1-a608ca87405f to bae69b96-6542-43be-a9c8-baae6b46a2e2 15:45:41.274 DEBUG [standby-run-23] StandbyClientSyncExecution.java:124 Inspecting segment bae69b96-6542-43be-a9c8-baae6b46a2e2 15:45:41.274 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 8a7b2823-22fe-4fd2-b975-265592eeca93 for references of segment bae69b96-6542-43be-a9c8-baae6b46a2e2 15:45:41.274 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:45:41.274 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment bae69b96-6542-43be-a9c8-baae6b46a2e2 for client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.274 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment bae69b96-6542-43be-a9c8-baae6b46a2e2 to client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.274 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:45:41.274 INFO [standby-run-23] StandbyClientSyncExecution.java:196 Copying data segment bae69b96-6542-43be-a9c8-baae6b46a2e2 from primary 15:45:41.290 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 8a7b2823-22fe-4fd2-b975-265592eeca93 for segment bae69b96-6542-43be-a9c8-baae6b46a2e2 15:45:41.290 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:45:41.290 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.290 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment bae69b96-6542-43be-a9c8-baae6b46a2e2 for client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.290 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 192 sent to client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.290 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment bae69b96-6542-43be-a9c8-baae6b46a2e2 to client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.290 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:45:41.290 INFO [standby-run-23] StandbyClientSyncExecution.java:196 Copying data segment 9e6fec1d-62b6-49e1-afb1-a608ca87405f from primary 15:45:41.290 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 8a7b2823-22fe-4fd2-b975-265592eeca93 for segment 9e6fec1d-62b6-49e1-afb1-a608ca87405f 15:45:41.290 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:45:41.290 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.290 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 9e6fec1d-62b6-49e1-afb1-a608ca87405f for client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.290 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 448 sent to client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.290 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 9e6fec1d-62b6-49e1-afb1-a608ca87405f to client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.290 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:45:41.290 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 8a7b2823-22fe-4fd2-b975-265592eeca93 for blob 0d3735c55c7c53ca8faaf03cabc5957b6ca9c187e570c4f3cadb8ec0f012a3d2#5242880 15:45:41.290 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 15:45:41.290 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get blob id' received from client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.290 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob 0d3735c55c7c53ca8faaf03cabc5957b6ca9c187e570c4f3cadb8ec0f012a3d2#5242880 for client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.290 DEBUG [primary-1] CommunicationObserver.java:105 Binary with size 5242880 sent to client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.290 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob 0d3735c55c7c53ca8faaf03cabc5957b6ca9c187e570c4f3cadb8ec0f012a3d2#5242880 to client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.306 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 0d3735c55c7c53ca8faaf03cabc5957b6ca9c187e570c4f3cadb8ec0f012a3d2#5242880 to client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.359 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:45:41.359 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 0d3735c55c7c53ca8faaf03cabc5957b6ca9c187e570c4f3cadb8ec0f012a3d2#5242880 15:45:41.359 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\0d3735c55c7c53ca8faaf03cabc5957b6ca9c187e570c4f3cadb8ec0f012a3d2#5242880.tmp 15:45:41.359 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 0d3735c55c7c53ca8faaf03cabc5957b6ca9c187e570c4f3cadb8ec0f012a3d2#5242880 to client 8a7b2823-22fe-4fd2-b975-265592eeca93 15:45:41.375 DEBUG [standby-run-23] StandbyClient.java:157 Channel closed 15:45:41.375 ERROR [standby-run-23] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to load remote blob 0d3735c55c7c53ca8faaf03cabc5957b6ca9c187e570c4f3cadb8ec0f012a3d2#5242880 at /root/server/#testBlob in 60ms. Please increase the timeout and try again. at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchAndStoreBlob(StandbyDiff.java:208) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:196) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:181) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:169) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeAdded(StandbyDiff.java:94) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeAdded(CancelableDiff.java:76) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:562) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:566) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:101) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.ExternalPrivateStoreIT.testSyncFailingDueToTooShortTimeout(ExternalPrivateStoreIT.java:86) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_60] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_60] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_60] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_60] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121) [surefire-booter-2.19.1.jar:2.19.1] 15:45:43.611 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:45:43.611 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:45:43.611 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:45:43.611 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:45:43.611 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:45:43.911 INFO [main] FileStore.java:471 TarMK closed: target\junit1105233665092179142\junit1365851795743929894 15:45:43.996 INFO [main] FileStore.java:471 TarMK closed: target\junit1105233665092179142\junit9185588256459436236 15:45:44.012 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit870315908339326517\junit4159668376225661389, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:45:44.027 INFO [main] FileStore.java:240 TarMK opened: target\junit870315908339326517\junit4159668376225661389 (mmap=false) 15:45:44.027 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit870315908339326517\junit3713090506342942328, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:45:44.027 INFO [main] FileStore.java:240 TarMK opened: target\junit870315908339326517\junit3713090506342942328 (mmap=false) 15:45:44.127 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:45:44.127 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:45:44.280 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x60c83dba]' 15:45:44.280 DEBUG [main] StandbyServer.java:249 Binding was successful 15:45:44.280 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 39165d1f-88e1-427c-9b04-49ccf330f458 for current head 15:45:44.280 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:52833 is allowed 15:45:44.280 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:45:44.280 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client 39165d1f-88e1-427c-9b04-49ccf330f458 15:45:44.280 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 39165d1f-88e1-427c-9b04-49ccf330f458 15:45:48.288 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 39165d1f-88e1-427c-9b04-49ccf330f458 io.netty.handler.timeout.ReadTimeoutException: null 15:45:48.290 ERROR [standby-run-24] StandbyClientSyncExecution.java:67 Unable to fetch remote head 15:45:48.290 DEBUG [standby-run-24] StandbyClient.java:157 Channel closed 15:45:49.684 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 5b42b7b5-4cd9-4546-a385-87a933230654.0000000c to client 39165d1f-88e1-427c-9b04-49ccf330f458 15:45:49.684 WARN [primary-1] ExceptionHandler.java:31 Exception caught on the server java.io.IOException: Eine bestehende Verbindung wurde softwaregesteuert durch den Hostcomputer abgebrochen at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[na:1.8.0_60] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) ~[na:1.8.0_60] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_60] at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_60] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_60] at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288) ~[netty-buffer-4.1.14.Final.jar:4.1.14.Final] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106) ~[netty-buffer-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:372) ~[netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) ~[netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] 15:45:50.495 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:45:50.495 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:45:50.495 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:45:50.495 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:45:50.495 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:45:50.580 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x7b8b7349]' 15:45:50.580 DEBUG [main] StandbyServer.java:249 Binding was successful 15:45:50.580 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d for current head 15:45:50.580 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:52874 is allowed 15:45:50.580 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:45:50.580 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.580 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.580 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 5b42b7b5-4cd9-4546-a385-87a933230654.0000000c to client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.596 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:45:50.596 DEBUG [standby-run-25] StandbyClientSyncExecution.java:103 Found missing segment 5b42b7b5-4cd9-4546-a385-87a933230654 15:45:50.596 DEBUG [standby-run-25] StandbyClientSyncExecution.java:124 Inspecting segment 5b42b7b5-4cd9-4546-a385-87a933230654 15:45:50.596 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d for references of segment 5b42b7b5-4cd9-4546-a385-87a933230654 15:45:50.596 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:45:50.596 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 5b42b7b5-4cd9-4546-a385-87a933230654 for client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.596 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 5b42b7b5-4cd9-4546-a385-87a933230654 to client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.596 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:45:50.596 DEBUG [standby-run-25] StandbyClientSyncExecution.java:184 Found reference from 5b42b7b5-4cd9-4546-a385-87a933230654 to 44186444-74bb-4d8a-ab35-869295936e84 15:45:50.596 DEBUG [standby-run-25] StandbyClientSyncExecution.java:124 Inspecting segment 44186444-74bb-4d8a-ab35-869295936e84 15:45:50.596 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d for references of segment 44186444-74bb-4d8a-ab35-869295936e84 15:45:50.596 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:45:50.596 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 44186444-74bb-4d8a-ab35-869295936e84 for client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.596 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 44186444-74bb-4d8a-ab35-869295936e84 to client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.596 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:45:50.596 INFO [standby-run-25] StandbyClientSyncExecution.java:196 Copying data segment 44186444-74bb-4d8a-ab35-869295936e84 from primary 15:45:50.596 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d for segment 44186444-74bb-4d8a-ab35-869295936e84 15:45:50.596 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:45:50.596 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.596 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 44186444-74bb-4d8a-ab35-869295936e84 for client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.596 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 192 sent to client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.596 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 44186444-74bb-4d8a-ab35-869295936e84 to client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.596 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:45:50.596 INFO [standby-run-25] StandbyClientSyncExecution.java:196 Copying data segment 5b42b7b5-4cd9-4546-a385-87a933230654 from primary 15:45:50.596 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d for segment 5b42b7b5-4cd9-4546-a385-87a933230654 15:45:50.596 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:45:50.596 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.596 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 5b42b7b5-4cd9-4546-a385-87a933230654 for client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.596 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 448 sent to client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.596 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 5b42b7b5-4cd9-4546-a385-87a933230654 to client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.596 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:45:50.596 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d for blob 4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880 15:45:50.596 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 15:45:50.596 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get blob id' received from client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.596 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob 4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880 for client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.596 DEBUG [primary-1] CommunicationObserver.java:105 Binary with size 5242880 sent to client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.596 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob 4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880 to client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.612 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880 to client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.643 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:45:50.643 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880 15:45:50.643 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880 to client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.643 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880.tmp 15:45:50.659 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880 to client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.659 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:45:50.659 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880 15:45:50.659 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880.tmp 15:45:50.681 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880 to client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.681 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:45:50.681 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880 15:45:50.681 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880.tmp 15:45:50.696 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:45:50.696 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880 15:45:50.696 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880 to client 9a8411e2-e5a6-4b17-aebc-0d7b8ff4736d 15:45:50.696 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880.tmp 15:45:50.743 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:45:50.743 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880 15:45:50.743 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880.tmp 15:45:50.743 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob 4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880 15:45:50.797 DEBUG [standby-run-25] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\4f67c5b35219df377c58ec14bb12d76ab94b5e2d53c140555be585a06119f14d#5242880.tmp 15:45:51.012 INFO [standby-run-25] StandbyClientSyncExecution.java:82 updated head state successfully: true in 416ms. 15:45:51.012 DEBUG [standby-run-25] StandbyClient.java:157 Channel closed 15:45:53.228 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:45:53.228 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:45:53.228 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:45:53.228 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:45:53.228 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:45:53.313 INFO [main] FileStore.java:471 TarMK closed: target\junit870315908339326517\junit3713090506342942328 15:45:53.344 INFO [main] FileStore.java:471 TarMK closed: target\junit870315908339326517\junit4159668376225661389 15:45:53.360 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1797098084298956562\junit7423200715580194945, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:45:53.377 INFO [main] FileStore.java:240 TarMK opened: target\junit1797098084298956562\junit7423200715580194945 (mmap=false) 15:45:53.381 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1797098084298956562\junit8948294261305420413, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:45:53.381 INFO [main] FileStore.java:240 TarMK opened: target\junit1797098084298956562\junit8948294261305420413 (mmap=false) 15:45:53.429 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:45:53.429 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:45:53.529 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=-1, skip=100,1 15:45:53.544 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x0458d060]' 15:45:53.560 DEBUG [main] StandbyServer.java:249 Binding was successful 15:45:53.745 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client f0b47c82-062f-42fc-a508-b79c1a54478c for current head 15:45:53.782 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:52960 is allowed 15:45:53.782 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:45:53.782 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:53.798 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:53.798 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head aeabddea-37e4-45b4-a733-35961af5d3fe.0000000c to client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:53.798 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:45:53.798 DEBUG [standby-run-26] StandbyClientSyncExecution.java:103 Found missing segment aeabddea-37e4-45b4-a733-35961af5d3fe 15:45:53.798 DEBUG [standby-run-26] StandbyClientSyncExecution.java:124 Inspecting segment aeabddea-37e4-45b4-a733-35961af5d3fe 15:45:53.798 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client f0b47c82-062f-42fc-a508-b79c1a54478c for references of segment aeabddea-37e4-45b4-a733-35961af5d3fe 15:45:53.798 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:45:53.798 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment aeabddea-37e4-45b4-a733-35961af5d3fe for client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:53.798 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment aeabddea-37e4-45b4-a733-35961af5d3fe to client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.801 DEBUG [standby-run-26] StandbyClient.java:157 Channel closed 15:45:58.801 ERROR [standby-run-26] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment aeabddea-37e4-45b4-a733-35961af5d3fe from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:340) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxySkippedBytes(DataStoreTestBase.java:289) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_60] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_60] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_60] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_60] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121) [surefire-booter-2.19.1.jar:2.19.1] 15:45:58.801 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=-1, skip=-1,0 15:45:58.817 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client f0b47c82-062f-42fc-a508-b79c1a54478c for current head 15:45:58.864 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:52978 is allowed 15:45:58.864 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 15:45:58.864 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get head' received from client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.864 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.864 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head aeabddea-37e4-45b4-a733-35961af5d3fe.0000000c to client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.864 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 15:45:58.864 DEBUG [standby-run-27] StandbyClientSyncExecution.java:103 Found missing segment aeabddea-37e4-45b4-a733-35961af5d3fe 15:45:58.864 DEBUG [standby-run-27] StandbyClientSyncExecution.java:124 Inspecting segment aeabddea-37e4-45b4-a733-35961af5d3fe 15:45:58.864 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client f0b47c82-062f-42fc-a508-b79c1a54478c for references of segment aeabddea-37e4-45b4-a733-35961af5d3fe 15:45:58.864 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:45:58.864 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment aeabddea-37e4-45b4-a733-35961af5d3fe for client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.864 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment aeabddea-37e4-45b4-a733-35961af5d3fe to client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.864 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:45:58.864 DEBUG [standby-run-27] StandbyClientSyncExecution.java:184 Found reference from aeabddea-37e4-45b4-a733-35961af5d3fe to e477fd97-9709-4369-ad3b-ee849de126d5 15:45:58.864 DEBUG [standby-run-27] StandbyClientSyncExecution.java:124 Inspecting segment e477fd97-9709-4369-ad3b-ee849de126d5 15:45:58.864 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client f0b47c82-062f-42fc-a508-b79c1a54478c for references of segment e477fd97-9709-4369-ad3b-ee849de126d5 15:45:58.864 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:45:58.864 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment e477fd97-9709-4369-ad3b-ee849de126d5 for client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.864 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment e477fd97-9709-4369-ad3b-ee849de126d5 to client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.880 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:45:58.880 INFO [standby-run-27] StandbyClientSyncExecution.java:196 Copying data segment e477fd97-9709-4369-ad3b-ee849de126d5 from primary 15:45:58.880 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client f0b47c82-062f-42fc-a508-b79c1a54478c for segment e477fd97-9709-4369-ad3b-ee849de126d5 15:45:58.881 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:45:58.881 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.881 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment e477fd97-9709-4369-ad3b-ee849de126d5 for client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.881 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 192 sent to client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.881 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment e477fd97-9709-4369-ad3b-ee849de126d5 to client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.882 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:45:58.882 INFO [standby-run-27] StandbyClientSyncExecution.java:196 Copying data segment aeabddea-37e4-45b4-a733-35961af5d3fe from primary 15:45:58.883 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client f0b47c82-062f-42fc-a508-b79c1a54478c for segment aeabddea-37e4-45b4-a733-35961af5d3fe 15:45:58.883 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:45:58.883 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.883 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment aeabddea-37e4-45b4-a733-35961af5d3fe for client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.884 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 448 sent to client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.884 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment aeabddea-37e4-45b4-a733-35961af5d3fe to client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.884 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:45:58.885 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client f0b47c82-062f-42fc-a508-b79c1a54478c for blob d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880 15:45:58.885 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 15:45:58.885 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get blob id' received from client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.885 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880 for client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.885 DEBUG [primary-2] CommunicationObserver.java:105 Binary with size 5242880 sent to client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.885 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880 to client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.885 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880 to client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.917 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880 to client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.917 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:45:58.917 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880 15:45:58.917 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880.tmp 15:45:58.948 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880 to client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.948 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:45:58.948 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880 15:45:58.948 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880.tmp 15:45:58.980 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880 to client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:58.986 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:45:58.986 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880 15:45:59.001 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880 to client f0b47c82-062f-42fc-a508-b79c1a54478c 15:45:59.001 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880.tmp 15:45:59.048 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:45:59.048 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880 15:45:59.048 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880.tmp 15:45:59.086 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:45:59.086 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880 15:45:59.086 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880.tmp 15:45:59.102 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880 15:45:59.148 DEBUG [standby-run-27] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\d174151af0e6d270a2192df9d038bb32bdf1bc9dc98bd9add479d9a524973f47#5242880.tmp 15:45:59.302 INFO [standby-run-27] StandbyClientSyncExecution.java:82 updated head state successfully: true in 438ms. 15:45:59.302 DEBUG [standby-run-27] StandbyClient.java:157 Channel closed 15:46:01.534 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:46:01.534 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:46:01.534 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:46:01.534 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:46:01.534 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:46:01.603 INFO [main] FileStore.java:471 TarMK closed: target\junit1797098084298956562\junit8948294261305420413 15:46:01.650 INFO [main] FileStore.java:471 TarMK closed: target\junit1797098084298956562\junit7423200715580194945 15:46:01.666 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit5529771478747058146\junit4767182206517556828, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:46:01.666 INFO [main] FileStore.java:240 TarMK opened: target\junit5529771478747058146\junit4767182206517556828 (mmap=false) 15:46:01.681 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit5529771478747058146\junit6542196249857111315, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:46:01.684 INFO [main] FileStore.java:240 TarMK opened: target\junit5529771478747058146\junit6542196249857111315 (mmap=false) 15:46:01.766 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:46:01.766 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:46:01.819 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x55394e9b]' 15:46:01.819 DEBUG [main] StandbyServer.java:249 Binding was successful 15:46:02.104 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 for current head 15:46:02.104 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:53049 is allowed 15:46:02.104 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:46:02.104 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.104 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.104 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head ab1c3da9-1ee3-47e3-a084-c6435732123c.0000000c to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.104 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:46:02.104 DEBUG [standby-run-28] StandbyClientSyncExecution.java:103 Found missing segment ab1c3da9-1ee3-47e3-a084-c6435732123c 15:46:02.104 DEBUG [standby-run-28] StandbyClientSyncExecution.java:124 Inspecting segment ab1c3da9-1ee3-47e3-a084-c6435732123c 15:46:02.104 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 for references of segment ab1c3da9-1ee3-47e3-a084-c6435732123c 15:46:02.104 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:46:02.104 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment ab1c3da9-1ee3-47e3-a084-c6435732123c for client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.104 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment ab1c3da9-1ee3-47e3-a084-c6435732123c to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.104 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:46:02.104 DEBUG [standby-run-28] StandbyClientSyncExecution.java:184 Found reference from ab1c3da9-1ee3-47e3-a084-c6435732123c to 2f02d638-3165-4b33-a691-700500f5637b 15:46:02.104 DEBUG [standby-run-28] StandbyClientSyncExecution.java:124 Inspecting segment 2f02d638-3165-4b33-a691-700500f5637b 15:46:02.120 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 for references of segment 2f02d638-3165-4b33-a691-700500f5637b 15:46:02.120 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:46:02.120 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 2f02d638-3165-4b33-a691-700500f5637b for client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.120 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 2f02d638-3165-4b33-a691-700500f5637b to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.120 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:46:02.120 INFO [standby-run-28] StandbyClientSyncExecution.java:196 Copying data segment 2f02d638-3165-4b33-a691-700500f5637b from primary 15:46:02.120 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 for segment 2f02d638-3165-4b33-a691-700500f5637b 15:46:02.120 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:46:02.120 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.120 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 2f02d638-3165-4b33-a691-700500f5637b for client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.120 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 192 sent to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.120 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 2f02d638-3165-4b33-a691-700500f5637b to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.120 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:02.120 INFO [standby-run-28] StandbyClientSyncExecution.java:196 Copying data segment ab1c3da9-1ee3-47e3-a084-c6435732123c from primary 15:46:02.120 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 for segment ab1c3da9-1ee3-47e3-a084-c6435732123c 15:46:02.120 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:46:02.120 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.120 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment ab1c3da9-1ee3-47e3-a084-c6435732123c for client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.120 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 448 sent to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.120 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment ab1c3da9-1ee3-47e3-a084-c6435732123c to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.120 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:02.135 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 for blob 4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880 15:46:02.135 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 15:46:02.135 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get blob id' received from client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.135 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob 4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880 for client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.135 DEBUG [primary-1] CommunicationObserver.java:105 Binary with size 5242880 sent to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.135 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob 4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880 to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.135 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880 to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.166 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:02.166 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880 15:46:02.166 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880 to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.166 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880.tmp 15:46:02.188 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880 to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.189 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:02.189 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880 15:46:02.189 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880.tmp 15:46:02.205 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:02.205 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880 15:46:02.205 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880 to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.205 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880.tmp 15:46:02.220 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:02.220 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880 15:46:02.220 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880 to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.220 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880.tmp 15:46:02.236 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:02.236 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880 15:46:02.236 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880.tmp 15:46:02.236 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob 4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880 15:46:02.289 DEBUG [standby-run-28] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\4baaa64585db0d3a06ca112da4693aaaceb78dff04002a10a5600b0fbd876a71#5242880.tmp 15:46:02.452 INFO [standby-run-28] StandbyClientSyncExecution.java:82 updated head state successfully: true in 348ms. 15:46:02.452 DEBUG [standby-run-28] StandbyClient.java:157 Channel closed 15:46:02.751 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 for current head 15:46:02.751 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:53050 is allowed 15:46:02.751 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 15:46:02.751 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get head' received from client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.751 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.751 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 978cfe4e-0712-4451-a310-428afaaf8e49.00000006 to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.751 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 15:46:02.751 DEBUG [standby-run-29] StandbyClientSyncExecution.java:103 Found missing segment 978cfe4e-0712-4451-a310-428afaaf8e49 15:46:02.751 DEBUG [standby-run-29] StandbyClientSyncExecution.java:124 Inspecting segment 978cfe4e-0712-4451-a310-428afaaf8e49 15:46:02.751 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 for references of segment 978cfe4e-0712-4451-a310-428afaaf8e49 15:46:02.751 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:46:02.751 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 978cfe4e-0712-4451-a310-428afaaf8e49 for client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.751 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 978cfe4e-0712-4451-a310-428afaaf8e49 to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.751 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:46:02.751 INFO [standby-run-29] StandbyClientSyncExecution.java:196 Copying data segment 978cfe4e-0712-4451-a310-428afaaf8e49 from primary 15:46:02.751 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 for segment 978cfe4e-0712-4451-a310-428afaaf8e49 15:46:02.767 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:46:02.767 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.767 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 978cfe4e-0712-4451-a310-428afaaf8e49 for client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.767 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 352 sent to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.767 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 978cfe4e-0712-4451-a310-428afaaf8e49 to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.767 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:02.767 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 for blob b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880 15:46:02.767 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 15:46:02.767 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get blob id' received from client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.767 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880 for client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.767 DEBUG [primary-2] CommunicationObserver.java:105 Binary with size 5242880 sent to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.767 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880 to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.767 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880 to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.789 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:02.789 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880 15:46:02.789 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880 to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.789 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880.tmp 15:46:02.820 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:02.820 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880 15:46:02.820 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880.tmp 15:46:02.820 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880 to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.851 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880 to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.867 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:02.867 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880 15:46:02.885 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880.tmp 15:46:02.890 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880 to client 73fe558f-c49e-4cfd-a8ff-bb3557a8fd10 15:46:02.905 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:02.905 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880 15:46:02.905 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880.tmp 15:46:02.921 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:02.921 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880 15:46:02.921 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880.tmp 15:46:02.921 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880 15:46:02.990 DEBUG [standby-run-29] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\b3ba831d72b696fdb62ea72681f34d96dc8a778a166815118898644afd7846e3#5242880.tmp 15:46:03.407 INFO [standby-run-29] StandbyClientSyncExecution.java:82 updated head state successfully: true in 656ms. 15:46:03.407 DEBUG [standby-run-29] StandbyClient.java:157 Channel closed 15:46:05.622 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:46:05.622 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:46:05.622 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:46:05.622 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:46:05.622 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:46:05.669 INFO [main] FileStore.java:471 TarMK closed: target\junit5529771478747058146\junit6542196249857111315 15:46:05.707 INFO [main] FileStore.java:471 TarMK closed: target\junit5529771478747058146\junit4767182206517556828 15:46:05.722 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit5035392730549752982\junit9040397228252900099, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:46:05.738 INFO [main] FileStore.java:240 TarMK opened: target\junit5035392730549752982\junit9040397228252900099 (mmap=false) 15:46:05.738 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit5035392730549752982\junit7530878099483105364, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:46:05.738 INFO [main] FileStore.java:240 TarMK opened: target\junit5035392730549752982\junit7530878099483105364 (mmap=false) 15:46:05.792 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:46:05.792 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:46:05.908 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x4a39cdbe]' 15:46:05.908 DEBUG [main] StandbyServer.java:249 Binding was successful 15:46:06.108 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 81673763-eaf2-4593-aa0e-0f76d600fe92 for current head 15:46:06.108 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:53119 is allowed 15:46:06.108 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:46:06.108 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.108 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.108 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head b5fdcdc5-d3c7-47e8-af48-02a797401e65.0000000c to client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.108 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:46:06.108 DEBUG [standby-run-30] StandbyClientSyncExecution.java:103 Found missing segment b5fdcdc5-d3c7-47e8-af48-02a797401e65 15:46:06.108 DEBUG [standby-run-30] StandbyClientSyncExecution.java:124 Inspecting segment b5fdcdc5-d3c7-47e8-af48-02a797401e65 15:46:06.108 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 81673763-eaf2-4593-aa0e-0f76d600fe92 for references of segment b5fdcdc5-d3c7-47e8-af48-02a797401e65 15:46:06.108 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:46:06.108 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment b5fdcdc5-d3c7-47e8-af48-02a797401e65 for client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.108 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment b5fdcdc5-d3c7-47e8-af48-02a797401e65 to client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.108 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:46:06.108 DEBUG [standby-run-30] StandbyClientSyncExecution.java:184 Found reference from b5fdcdc5-d3c7-47e8-af48-02a797401e65 to 282066b9-a569-4a10-a522-9fb52c04ff26 15:46:06.108 DEBUG [standby-run-30] StandbyClientSyncExecution.java:124 Inspecting segment 282066b9-a569-4a10-a522-9fb52c04ff26 15:46:06.108 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 81673763-eaf2-4593-aa0e-0f76d600fe92 for references of segment 282066b9-a569-4a10-a522-9fb52c04ff26 15:46:06.123 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:46:06.123 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 282066b9-a569-4a10-a522-9fb52c04ff26 for client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.123 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 282066b9-a569-4a10-a522-9fb52c04ff26 to client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.123 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:46:06.123 INFO [standby-run-30] StandbyClientSyncExecution.java:196 Copying data segment 282066b9-a569-4a10-a522-9fb52c04ff26 from primary 15:46:06.123 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 81673763-eaf2-4593-aa0e-0f76d600fe92 for segment 282066b9-a569-4a10-a522-9fb52c04ff26 15:46:06.123 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:46:06.123 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.123 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 282066b9-a569-4a10-a522-9fb52c04ff26 for client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.123 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 192 sent to client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.123 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 282066b9-a569-4a10-a522-9fb52c04ff26 to client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.123 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:06.123 INFO [standby-run-30] StandbyClientSyncExecution.java:196 Copying data segment b5fdcdc5-d3c7-47e8-af48-02a797401e65 from primary 15:46:06.123 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 81673763-eaf2-4593-aa0e-0f76d600fe92 for segment b5fdcdc5-d3c7-47e8-af48-02a797401e65 15:46:06.123 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:46:06.123 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.123 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment b5fdcdc5-d3c7-47e8-af48-02a797401e65 for client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.123 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 448 sent to client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.123 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment b5fdcdc5-d3c7-47e8-af48-02a797401e65 to client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.123 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:06.139 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 81673763-eaf2-4593-aa0e-0f76d600fe92 for blob 234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880 15:46:06.139 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 15:46:06.139 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get blob id' received from client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.139 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob 234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880 for client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.139 DEBUG [primary-1] CommunicationObserver.java:105 Binary with size 5242880 sent to client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.139 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob 234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880 to client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.139 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880 to client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.192 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:06.192 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880 15:46:06.192 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880 to client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.192 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880.tmp 15:46:06.208 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:06.208 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880 15:46:06.208 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880 to client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.208 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880.tmp 15:46:06.223 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:06.223 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880 15:46:06.223 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880 to client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.223 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880.tmp 15:46:06.239 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:06.239 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880 15:46:06.239 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880 to client 81673763-eaf2-4593-aa0e-0f76d600fe92 15:46:06.239 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880.tmp 15:46:06.270 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:06.270 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880 15:46:06.270 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880.tmp 15:46:06.270 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob 234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880 15:46:06.323 DEBUG [standby-run-30] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\234584b5d8fab7c8f010ae973175a76cb4d42f4b669b4eaabc1a2350ddf5980f#5242880.tmp 15:46:06.439 INFO [standby-run-30] StandbyClientSyncExecution.java:82 updated head state successfully: true in 331ms. 15:46:06.439 DEBUG [standby-run-30] StandbyClient.java:157 Channel closed 15:46:08.671 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:46:08.671 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:46:08.671 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:46:08.671 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:46:08.671 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:46:08.740 INFO [main] FileStore.java:471 TarMK closed: target\junit5035392730549752982\junit7530878099483105364 15:46:08.771 INFO [main] FileStore.java:471 TarMK closed: target\junit5035392730549752982\junit9040397228252900099 15:46:08.793 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7447044532776233102\junit544792357884221034, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:46:08.793 INFO [main] FileStore.java:240 TarMK opened: target\junit7447044532776233102\junit544792357884221034 (mmap=false) 15:46:08.809 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7447044532776233102\junit2202162313515509780, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:46:08.809 INFO [main] FileStore.java:240 TarMK opened: target\junit7447044532776233102\junit2202162313515509780 (mmap=false) 15:46:08.856 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:46:08.856 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:46:08.956 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=150000, skip=0,0 15:46:08.993 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xecd41592]' 15:46:08.994 DEBUG [main] StandbyServer.java:249 Binding was successful 15:46:09.340 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c for current head 15:46:09.389 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:53207 is allowed 15:46:09.389 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:46:09.390 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.390 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.390 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 44f2b1be-1f97-43a7-a8bf-afd8f6ae267d.0000000c to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.391 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:46:09.391 DEBUG [standby-run-31] StandbyClientSyncExecution.java:103 Found missing segment 44f2b1be-1f97-43a7-a8bf-afd8f6ae267d 15:46:09.391 DEBUG [standby-run-31] StandbyClientSyncExecution.java:124 Inspecting segment 44f2b1be-1f97-43a7-a8bf-afd8f6ae267d 15:46:09.394 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c for references of segment 44f2b1be-1f97-43a7-a8bf-afd8f6ae267d 15:46:09.394 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:46:09.394 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 44f2b1be-1f97-43a7-a8bf-afd8f6ae267d for client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.394 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 44f2b1be-1f97-43a7-a8bf-afd8f6ae267d to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.394 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:46:09.394 DEBUG [standby-run-31] StandbyClientSyncExecution.java:184 Found reference from 44f2b1be-1f97-43a7-a8bf-afd8f6ae267d to 1ab31ee9-5e1a-44dc-a661-34b24168e747 15:46:09.394 DEBUG [standby-run-31] StandbyClientSyncExecution.java:124 Inspecting segment 1ab31ee9-5e1a-44dc-a661-34b24168e747 15:46:09.394 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c for references of segment 1ab31ee9-5e1a-44dc-a661-34b24168e747 15:46:09.394 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:46:09.394 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 1ab31ee9-5e1a-44dc-a661-34b24168e747 for client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.394 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 1ab31ee9-5e1a-44dc-a661-34b24168e747 to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.394 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:46:09.394 INFO [standby-run-31] StandbyClientSyncExecution.java:196 Copying data segment 1ab31ee9-5e1a-44dc-a661-34b24168e747 from primary 15:46:09.394 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c for segment 1ab31ee9-5e1a-44dc-a661-34b24168e747 15:46:09.394 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:46:09.394 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.394 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 1ab31ee9-5e1a-44dc-a661-34b24168e747 for client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.394 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 192 sent to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.394 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 1ab31ee9-5e1a-44dc-a661-34b24168e747 to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.409 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:09.409 INFO [standby-run-31] StandbyClientSyncExecution.java:196 Copying data segment 44f2b1be-1f97-43a7-a8bf-afd8f6ae267d from primary 15:46:09.409 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c for segment 44f2b1be-1f97-43a7-a8bf-afd8f6ae267d 15:46:09.409 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:46:09.409 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.409 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 44f2b1be-1f97-43a7-a8bf-afd8f6ae267d for client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.409 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 448 sent to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.409 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 44f2b1be-1f97-43a7-a8bf-afd8f6ae267d to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.409 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:09.409 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c for blob a13d29ae2026092e5a27efac16b334ce8b0b70a9f407f85361edc4125a73e9dd#5242880 15:46:09.409 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 15:46:09.409 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get blob id' received from client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.409 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob a13d29ae2026092e5a27efac16b334ce8b0b70a9f407f85361edc4125a73e9dd#5242880 for client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.409 DEBUG [primary-1] CommunicationObserver.java:105 Binary with size 5242880 sent to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.409 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob a13d29ae2026092e5a27efac16b334ce8b0b70a9f407f85361edc4125a73e9dd#5242880 to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.409 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob a13d29ae2026092e5a27efac16b334ce8b0b70a9f407f85361edc4125a73e9dd#5242880 to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.440 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c io.netty.handler.codec.compression.DecompressionException: mismatching checksum: a282120b (expected: a28287dd) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] 15:46:09.472 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob a13d29ae2026092e5a27efac16b334ce8b0b70a9f407f85361edc4125a73e9dd#5242880 to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.509 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob a13d29ae2026092e5a27efac16b334ce8b0b70a9f407f85361edc4125a73e9dd#5242880 to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.556 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob a13d29ae2026092e5a27efac16b334ce8b0b70a9f407f85361edc4125a73e9dd#5242880 to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:09.610 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob a13d29ae2026092e5a27efac16b334ce8b0b70a9f407f85361edc4125a73e9dd#5242880 to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:14.427 DEBUG [standby-run-31] StandbyClient.java:157 Channel closed 15:46:14.427 ERROR [standby-run-31] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to load remote blob a13d29ae2026092e5a27efac16b334ce8b0b70a9f407f85361edc4125a73e9dd#5242880 at /root/server/#testBlob in 5000ms. Please increase the timeout and try again. at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchAndStoreBlob(StandbyDiff.java:208) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:196) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:181) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:169) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeAdded(StandbyDiff.java:94) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeAdded(CancelableDiff.java:76) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:562) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:566) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:101) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:340) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByteChange2(DataStoreTestBase.java:319) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_60] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_60] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_60] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_60] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121) [surefire-booter-2.19.1.jar:2.19.1] 15:46:14.427 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=-1, skip=-1,0 15:46:14.627 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c for current head 15:46:14.674 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:53225 is allowed 15:46:14.674 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 15:46:14.674 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get head' received from client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:14.674 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:14.674 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 0f3860cd-d94d-47f3-a1f0-3b29bced5a12.00000006 to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:14.674 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 15:46:14.674 DEBUG [standby-run-32] StandbyClientSyncExecution.java:103 Found missing segment 0f3860cd-d94d-47f3-a1f0-3b29bced5a12 15:46:14.674 DEBUG [standby-run-32] StandbyClientSyncExecution.java:124 Inspecting segment 0f3860cd-d94d-47f3-a1f0-3b29bced5a12 15:46:14.674 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c for references of segment 0f3860cd-d94d-47f3-a1f0-3b29bced5a12 15:46:14.674 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:46:14.674 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 0f3860cd-d94d-47f3-a1f0-3b29bced5a12 for client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:14.674 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 0f3860cd-d94d-47f3-a1f0-3b29bced5a12 to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:14.674 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:46:14.674 INFO [standby-run-32] StandbyClientSyncExecution.java:196 Copying data segment 0f3860cd-d94d-47f3-a1f0-3b29bced5a12 from primary 15:46:14.674 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c for segment 0f3860cd-d94d-47f3-a1f0-3b29bced5a12 15:46:14.674 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:46:14.674 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:14.674 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 0f3860cd-d94d-47f3-a1f0-3b29bced5a12 for client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:14.674 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 352 sent to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:14.674 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 0f3860cd-d94d-47f3-a1f0-3b29bced5a12 to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:14.674 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:14.674 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c for blob a10760ebadbb2bb97aa8c1fd6348c33a07a287268475abcffe18e312d010576c#2097152 15:46:14.674 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 15:46:14.674 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get blob id' received from client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:14.674 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob a10760ebadbb2bb97aa8c1fd6348c33a07a287268475abcffe18e312d010576c#2097152 for client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:14.674 DEBUG [primary-2] CommunicationObserver.java:105 Binary with size 2097152 sent to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:14.674 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob a10760ebadbb2bb97aa8c1fd6348c33a07a287268475abcffe18e312d010576c#2097152 to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:14.674 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/2 of size 1048576 from blob a10760ebadbb2bb97aa8c1fd6348c33a07a287268475abcffe18e312d010576c#2097152 to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:14.711 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/2 of size 1048576 from blob a10760ebadbb2bb97aa8c1fd6348c33a07a287268475abcffe18e312d010576c#2097152 to client f25e1f65-35c4-4dd3-b571-6eaaa50d3f0c 15:46:14.711 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:14.711 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/2 of size 1048576 from blob a10760ebadbb2bb97aa8c1fd6348c33a07a287268475abcffe18e312d010576c#2097152 15:46:14.727 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\a10760ebadbb2bb97aa8c1fd6348c33a07a287268475abcffe18e312d010576c#2097152.tmp 15:46:14.758 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:14.774 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/2 of size 1048576 from blob a10760ebadbb2bb97aa8c1fd6348c33a07a287268475abcffe18e312d010576c#2097152 15:46:14.774 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\a10760ebadbb2bb97aa8c1fd6348c33a07a287268475abcffe18e312d010576c#2097152.tmp 15:46:14.774 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob a10760ebadbb2bb97aa8c1fd6348c33a07a287268475abcffe18e312d010576c#2097152 15:46:14.796 DEBUG [standby-run-32] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\a10760ebadbb2bb97aa8c1fd6348c33a07a287268475abcffe18e312d010576c#2097152.tmp 15:46:14.911 INFO [standby-run-32] StandbyClientSyncExecution.java:82 updated head state successfully: true in 237ms. 15:46:14.927 DEBUG [standby-run-32] StandbyClient.java:157 Channel closed 15:46:17.259 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:46:17.259 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:46:17.259 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:46:17.259 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:46:17.259 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:46:17.359 INFO [main] FileStore.java:471 TarMK closed: target\junit7447044532776233102\junit2202162313515509780 15:46:17.397 INFO [main] FileStore.java:471 TarMK closed: target\junit7447044532776233102\junit544792357884221034 15:46:17.428 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit9067909772135242338\junit7650660574508625459, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:46:17.428 INFO [main] FileStore.java:240 TarMK opened: target\junit9067909772135242338\junit7650660574508625459 (mmap=false) 15:46:17.443 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit9067909772135242338\junit5445220661370973239, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:46:17.443 INFO [main] FileStore.java:240 TarMK opened: target\junit9067909772135242338\junit5445220661370973239 (mmap=false) 15:46:17.512 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:46:17.512 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:46:17.628 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=150, skip=0,0 15:46:17.660 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xc4b75f3f]' 15:46:17.660 DEBUG [main] StandbyServer.java:249 Binding was successful 15:46:17.813 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client b90f53b9-69ad-4f2c-948d-896e69942181 for current head 15:46:17.860 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:53314 is allowed 15:46:17.860 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:46:17.860 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:17.860 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:17.860 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head e673066c-6eb7-4760-a967-ad0011f1d4d0.0000000c to client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:17.860 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:46:17.860 DEBUG [standby-run-33] StandbyClientSyncExecution.java:103 Found missing segment e673066c-6eb7-4760-a967-ad0011f1d4d0 15:46:17.860 DEBUG [standby-run-33] StandbyClientSyncExecution.java:124 Inspecting segment e673066c-6eb7-4760-a967-ad0011f1d4d0 15:46:17.860 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client b90f53b9-69ad-4f2c-948d-896e69942181 for references of segment e673066c-6eb7-4760-a967-ad0011f1d4d0 15:46:17.860 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:46:17.860 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment e673066c-6eb7-4760-a967-ad0011f1d4d0 for client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:17.860 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment e673066c-6eb7-4760-a967-ad0011f1d4d0 to client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:17.860 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client b90f53b9-69ad-4f2c-948d-896e69942181 io.netty.handler.codec.compression.DecompressionException: mismatching checksum: a28232d3 (expected: 98d75388) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] 15:46:22.870 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client b90f53b9-69ad-4f2c-948d-896e69942181 io.netty.handler.timeout.ReadTimeoutException: null 15:46:22.870 DEBUG [standby-run-33] StandbyClient.java:157 Channel closed 15:46:22.870 ERROR [standby-run-33] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment e673066c-6eb7-4760-a967-ad0011f1d4d0 from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:340) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByteChange(DataStoreTestBase.java:314) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_60] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_60] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_60] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_60] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121) [surefire-booter-2.19.1.jar:2.19.1] 15:46:22.870 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=-1, skip=-1,0 15:46:23.086 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client b90f53b9-69ad-4f2c-948d-896e69942181 for current head 15:46:23.139 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:53336 is allowed 15:46:23.139 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 15:46:23.139 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get head' received from client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.139 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.139 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 2e3dab46-83ed-46a4-ad9d-84733e1667ba.00000006 to client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.139 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 15:46:23.139 DEBUG [standby-run-34] StandbyClientSyncExecution.java:103 Found missing segment 2e3dab46-83ed-46a4-ad9d-84733e1667ba 15:46:23.139 DEBUG [standby-run-34] StandbyClientSyncExecution.java:124 Inspecting segment 2e3dab46-83ed-46a4-ad9d-84733e1667ba 15:46:23.139 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client b90f53b9-69ad-4f2c-948d-896e69942181 for references of segment 2e3dab46-83ed-46a4-ad9d-84733e1667ba 15:46:23.139 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:46:23.139 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 2e3dab46-83ed-46a4-ad9d-84733e1667ba for client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.139 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 2e3dab46-83ed-46a4-ad9d-84733e1667ba to client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.139 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:46:23.139 DEBUG [standby-run-34] StandbyClientSyncExecution.java:184 Found reference from 2e3dab46-83ed-46a4-ad9d-84733e1667ba to e673066c-6eb7-4760-a967-ad0011f1d4d0 15:46:23.139 DEBUG [standby-run-34] StandbyClientSyncExecution.java:184 Found reference from 2e3dab46-83ed-46a4-ad9d-84733e1667ba to 6cb53f09-514c-4580-ae8b-1a7cc3886066 15:46:23.139 DEBUG [standby-run-34] StandbyClientSyncExecution.java:124 Inspecting segment e673066c-6eb7-4760-a967-ad0011f1d4d0 15:46:23.139 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client b90f53b9-69ad-4f2c-948d-896e69942181 for references of segment e673066c-6eb7-4760-a967-ad0011f1d4d0 15:46:23.139 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:46:23.139 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment e673066c-6eb7-4760-a967-ad0011f1d4d0 for client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.139 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment e673066c-6eb7-4760-a967-ad0011f1d4d0 to client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.139 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:46:23.139 DEBUG [standby-run-34] StandbyClientSyncExecution.java:124 Inspecting segment 6cb53f09-514c-4580-ae8b-1a7cc3886066 15:46:23.139 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client b90f53b9-69ad-4f2c-948d-896e69942181 for references of segment 6cb53f09-514c-4580-ae8b-1a7cc3886066 15:46:23.139 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:46:23.139 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 6cb53f09-514c-4580-ae8b-1a7cc3886066 for client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.139 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 6cb53f09-514c-4580-ae8b-1a7cc3886066 to client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.139 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:46:23.139 INFO [standby-run-34] StandbyClientSyncExecution.java:196 Copying data segment 6cb53f09-514c-4580-ae8b-1a7cc3886066 from primary 15:46:23.139 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client b90f53b9-69ad-4f2c-948d-896e69942181 for segment 6cb53f09-514c-4580-ae8b-1a7cc3886066 15:46:23.139 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:46:23.139 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.139 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 6cb53f09-514c-4580-ae8b-1a7cc3886066 for client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.139 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 192 sent to client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.139 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 6cb53f09-514c-4580-ae8b-1a7cc3886066 to client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.139 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:23.139 INFO [standby-run-34] StandbyClientSyncExecution.java:196 Copying data segment e673066c-6eb7-4760-a967-ad0011f1d4d0 from primary 15:46:23.139 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client b90f53b9-69ad-4f2c-948d-896e69942181 for segment e673066c-6eb7-4760-a967-ad0011f1d4d0 15:46:23.139 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:46:23.139 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.139 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment e673066c-6eb7-4760-a967-ad0011f1d4d0 for client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.139 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 448 sent to client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.139 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment e673066c-6eb7-4760-a967-ad0011f1d4d0 to client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.139 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:23.139 INFO [standby-run-34] StandbyClientSyncExecution.java:196 Copying data segment 2e3dab46-83ed-46a4-ad9d-84733e1667ba from primary 15:46:23.139 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client b90f53b9-69ad-4f2c-948d-896e69942181 for segment 2e3dab46-83ed-46a4-ad9d-84733e1667ba 15:46:23.155 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:46:23.155 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.155 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 2e3dab46-83ed-46a4-ad9d-84733e1667ba for client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.155 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 352 sent to client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.155 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 2e3dab46-83ed-46a4-ad9d-84733e1667ba to client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.155 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:23.155 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client b90f53b9-69ad-4f2c-948d-896e69942181 for blob 696b30c4cefe61800589465c8d7e8d80f5a9415159bfae35e9aa81a73cd9d618#2097152 15:46:23.155 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 15:46:23.155 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get blob id' received from client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.155 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 696b30c4cefe61800589465c8d7e8d80f5a9415159bfae35e9aa81a73cd9d618#2097152 for client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.155 DEBUG [primary-2] CommunicationObserver.java:105 Binary with size 2097152 sent to client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.155 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 696b30c4cefe61800589465c8d7e8d80f5a9415159bfae35e9aa81a73cd9d618#2097152 to client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.155 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/2 of size 1048576 from blob 696b30c4cefe61800589465c8d7e8d80f5a9415159bfae35e9aa81a73cd9d618#2097152 to client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.186 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/2 of size 1048576 from blob 696b30c4cefe61800589465c8d7e8d80f5a9415159bfae35e9aa81a73cd9d618#2097152 to client b90f53b9-69ad-4f2c-948d-896e69942181 15:46:23.203 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:23.203 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/2 of size 1048576 from blob 696b30c4cefe61800589465c8d7e8d80f5a9415159bfae35e9aa81a73cd9d618#2097152 15:46:23.206 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\696b30c4cefe61800589465c8d7e8d80f5a9415159bfae35e9aa81a73cd9d618#2097152.tmp 15:46:23.239 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:23.239 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/2 of size 1048576 from blob 696b30c4cefe61800589465c8d7e8d80f5a9415159bfae35e9aa81a73cd9d618#2097152 15:46:23.239 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\696b30c4cefe61800589465c8d7e8d80f5a9415159bfae35e9aa81a73cd9d618#2097152.tmp 15:46:23.239 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob 696b30c4cefe61800589465c8d7e8d80f5a9415159bfae35e9aa81a73cd9d618#2097152 15:46:23.271 DEBUG [standby-run-34] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\696b30c4cefe61800589465c8d7e8d80f5a9415159bfae35e9aa81a73cd9d618#2097152.tmp 15:46:23.386 INFO [standby-run-34] StandbyClientSyncExecution.java:82 updated head state successfully: true in 247ms. 15:46:23.386 DEBUG [standby-run-34] StandbyClient.java:157 Channel closed 15:46:25.702 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:46:25.703 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:46:25.704 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:46:25.705 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:46:25.708 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:46:25.740 INFO [main] FileStore.java:471 TarMK closed: target\junit9067909772135242338\junit5445220661370973239 15:46:25.771 INFO [main] FileStore.java:471 TarMK closed: target\junit9067909772135242338\junit7650660574508625459 15:46:25.804 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit3231619518313213710\junit5374215729641058742, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:46:25.806 INFO [main] FileStore.java:240 TarMK opened: target\junit3231619518313213710\junit5374215729641058742 (mmap=false) 15:46:25.808 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit3231619518313213710\junit7721803718779725458, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:46:25.808 INFO [main] FileStore.java:240 TarMK opened: target\junit3231619518313213710\junit7721803718779725458 (mmap=false) 15:46:25.855 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:46:25.855 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:46:25.956 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=-1, skip=100,1 15:46:25.987 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x1805fcfa]' 15:46:25.987 DEBUG [main] StandbyServer.java:249 Binding was successful 15:46:26.241 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client fa733056-27d2-4311-badb-c9a654138f3a for current head 15:46:26.288 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:53422 is allowed 15:46:26.288 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:46:26.288 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client fa733056-27d2-4311-badb-c9a654138f3a 15:46:26.288 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client fa733056-27d2-4311-badb-c9a654138f3a 15:46:26.288 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 249ad0cc-8b88-4391-a0d1-211833b6ac35.0000000c to client fa733056-27d2-4311-badb-c9a654138f3a 15:46:26.288 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:46:26.288 DEBUG [standby-run-35] StandbyClientSyncExecution.java:103 Found missing segment 249ad0cc-8b88-4391-a0d1-211833b6ac35 15:46:26.288 DEBUG [standby-run-35] StandbyClientSyncExecution.java:124 Inspecting segment 249ad0cc-8b88-4391-a0d1-211833b6ac35 15:46:26.288 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client fa733056-27d2-4311-badb-c9a654138f3a for references of segment 249ad0cc-8b88-4391-a0d1-211833b6ac35 15:46:26.288 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:46:26.288 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 249ad0cc-8b88-4391-a0d1-211833b6ac35 for client fa733056-27d2-4311-badb-c9a654138f3a 15:46:26.288 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 249ad0cc-8b88-4391-a0d1-211833b6ac35 to client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.309 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client fa733056-27d2-4311-badb-c9a654138f3a io.netty.handler.timeout.ReadTimeoutException: null 15:46:31.310 DEBUG [standby-run-35] StandbyClient.java:157 Channel closed 15:46:31.311 ERROR [standby-run-35] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment 249ad0cc-8b88-4391-a0d1-211833b6ac35 from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:340) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxySkippedBytesIntermediateChange(DataStoreTestBase.java:294) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_60] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_60] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_60] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_60] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121) [surefire-booter-2.19.1.jar:2.19.1] 15:46:31.311 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=-1, skip=-1,0 15:46:31.747 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client fa733056-27d2-4311-badb-c9a654138f3a for current head 15:46:31.763 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:53492 is allowed 15:46:31.763 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 15:46:31.763 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get head' received from client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.763 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.763 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 56baedc1-8a4b-49e8-a48c-93483280ac65.00000006 to client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.763 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 15:46:31.763 DEBUG [standby-run-36] StandbyClientSyncExecution.java:103 Found missing segment 56baedc1-8a4b-49e8-a48c-93483280ac65 15:46:31.763 DEBUG [standby-run-36] StandbyClientSyncExecution.java:124 Inspecting segment 56baedc1-8a4b-49e8-a48c-93483280ac65 15:46:31.763 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client fa733056-27d2-4311-badb-c9a654138f3a for references of segment 56baedc1-8a4b-49e8-a48c-93483280ac65 15:46:31.763 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:46:31.778 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 56baedc1-8a4b-49e8-a48c-93483280ac65 for client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.778 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 56baedc1-8a4b-49e8-a48c-93483280ac65 to client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.778 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:46:31.778 DEBUG [standby-run-36] StandbyClientSyncExecution.java:184 Found reference from 56baedc1-8a4b-49e8-a48c-93483280ac65 to 249ad0cc-8b88-4391-a0d1-211833b6ac35 15:46:31.778 DEBUG [standby-run-36] StandbyClientSyncExecution.java:184 Found reference from 56baedc1-8a4b-49e8-a48c-93483280ac65 to 858fdb82-f7d6-40ad-ab11-40b246210f27 15:46:31.778 DEBUG [standby-run-36] StandbyClientSyncExecution.java:124 Inspecting segment 249ad0cc-8b88-4391-a0d1-211833b6ac35 15:46:31.778 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client fa733056-27d2-4311-badb-c9a654138f3a for references of segment 249ad0cc-8b88-4391-a0d1-211833b6ac35 15:46:31.778 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:46:31.778 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 249ad0cc-8b88-4391-a0d1-211833b6ac35 for client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.778 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 249ad0cc-8b88-4391-a0d1-211833b6ac35 to client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.778 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:46:31.778 DEBUG [standby-run-36] StandbyClientSyncExecution.java:124 Inspecting segment 858fdb82-f7d6-40ad-ab11-40b246210f27 15:46:31.778 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client fa733056-27d2-4311-badb-c9a654138f3a for references of segment 858fdb82-f7d6-40ad-ab11-40b246210f27 15:46:31.778 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:46:31.778 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 858fdb82-f7d6-40ad-ab11-40b246210f27 for client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.778 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 858fdb82-f7d6-40ad-ab11-40b246210f27 to client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.778 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:46:31.778 INFO [standby-run-36] StandbyClientSyncExecution.java:196 Copying data segment 858fdb82-f7d6-40ad-ab11-40b246210f27 from primary 15:46:31.778 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client fa733056-27d2-4311-badb-c9a654138f3a for segment 858fdb82-f7d6-40ad-ab11-40b246210f27 15:46:31.778 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:46:31.778 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.778 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 858fdb82-f7d6-40ad-ab11-40b246210f27 for client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.778 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 192 sent to client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.794 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 858fdb82-f7d6-40ad-ab11-40b246210f27 to client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.794 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:31.794 INFO [standby-run-36] StandbyClientSyncExecution.java:196 Copying data segment 249ad0cc-8b88-4391-a0d1-211833b6ac35 from primary 15:46:31.794 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client fa733056-27d2-4311-badb-c9a654138f3a for segment 249ad0cc-8b88-4391-a0d1-211833b6ac35 15:46:31.794 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:46:31.794 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.794 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 249ad0cc-8b88-4391-a0d1-211833b6ac35 for client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.794 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 448 sent to client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.794 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 249ad0cc-8b88-4391-a0d1-211833b6ac35 to client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.794 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:31.794 INFO [standby-run-36] StandbyClientSyncExecution.java:196 Copying data segment 56baedc1-8a4b-49e8-a48c-93483280ac65 from primary 15:46:31.794 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client fa733056-27d2-4311-badb-c9a654138f3a for segment 56baedc1-8a4b-49e8-a48c-93483280ac65 15:46:31.794 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:46:31.794 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.794 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 56baedc1-8a4b-49e8-a48c-93483280ac65 for client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.794 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 352 sent to client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.794 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 56baedc1-8a4b-49e8-a48c-93483280ac65 to client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.794 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:31.794 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client fa733056-27d2-4311-badb-c9a654138f3a for blob 15f024e866f6e921ccdaeb216af1dce204e95bafe54b0bd581c4f4ea28e95b04#2097152 15:46:31.794 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 15:46:31.794 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get blob id' received from client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.794 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 15f024e866f6e921ccdaeb216af1dce204e95bafe54b0bd581c4f4ea28e95b04#2097152 for client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.794 DEBUG [primary-2] CommunicationObserver.java:105 Binary with size 2097152 sent to client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.794 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 15f024e866f6e921ccdaeb216af1dce204e95bafe54b0bd581c4f4ea28e95b04#2097152 to client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.816 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/2 of size 1048576 from blob 15f024e866f6e921ccdaeb216af1dce204e95bafe54b0bd581c4f4ea28e95b04#2097152 to client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.847 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/2 of size 1048576 from blob 15f024e866f6e921ccdaeb216af1dce204e95bafe54b0bd581c4f4ea28e95b04#2097152 to client fa733056-27d2-4311-badb-c9a654138f3a 15:46:31.847 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:31.847 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/2 of size 1048576 from blob 15f024e866f6e921ccdaeb216af1dce204e95bafe54b0bd581c4f4ea28e95b04#2097152 15:46:31.863 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\15f024e866f6e921ccdaeb216af1dce204e95bafe54b0bd581c4f4ea28e95b04#2097152.tmp 15:46:31.935 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:31.936 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/2 of size 1048576 from blob 15f024e866f6e921ccdaeb216af1dce204e95bafe54b0bd581c4f4ea28e95b04#2097152 15:46:31.939 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\15f024e866f6e921ccdaeb216af1dce204e95bafe54b0bd581c4f4ea28e95b04#2097152.tmp 15:46:31.977 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob 15f024e866f6e921ccdaeb216af1dce204e95bafe54b0bd581c4f4ea28e95b04#2097152 15:46:32.024 DEBUG [standby-run-36] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\15f024e866f6e921ccdaeb216af1dce204e95bafe54b0bd581c4f4ea28e95b04#2097152.tmp 15:46:32.172 INFO [standby-run-36] StandbyClientSyncExecution.java:82 updated head state successfully: true in 409ms. 15:46:32.173 DEBUG [standby-run-36] StandbyClient.java:157 Channel closed 15:46:34.407 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:46:34.407 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:46:34.408 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:46:34.408 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:46:34.410 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:46:34.475 INFO [main] FileStore.java:471 TarMK closed: target\junit3231619518313213710\junit7721803718779725458 15:46:34.512 INFO [main] FileStore.java:471 TarMK closed: target\junit3231619518313213710\junit5374215729641058742 15:46:34.559 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit4451559270981989588\junit3939919258085895365, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:46:34.559 INFO [main] FileStore.java:240 TarMK opened: target\junit4451559270981989588\junit3939919258085895365 (mmap=false) 15:46:34.575 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit4451559270981989588\junit5485841426612374562, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:46:34.575 INFO [main] FileStore.java:240 TarMK opened: target\junit4451559270981989588\junit5485841426612374562 (mmap=false) 15:46:34.628 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:46:34.628 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:46:34.728 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=150, skip=0,0 15:46:34.760 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x28588009]' 15:46:34.775 DEBUG [main] StandbyServer.java:249 Binding was successful 15:46:35.460 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 148d14a6-388c-4371-b59f-5863c4b74903 for current head 15:46:35.510 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:53602 is allowed 15:46:35.510 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:46:35.510 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:35.511 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:35.511 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head d1951373-a638-4d2b-af06-87dd7beb0ebc.0000000c to client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:35.512 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:46:35.512 DEBUG [standby-run-37] StandbyClientSyncExecution.java:103 Found missing segment d1951373-a638-4d2b-af06-87dd7beb0ebc 15:46:35.512 DEBUG [standby-run-37] StandbyClientSyncExecution.java:124 Inspecting segment d1951373-a638-4d2b-af06-87dd7beb0ebc 15:46:35.513 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 148d14a6-388c-4371-b59f-5863c4b74903 for references of segment d1951373-a638-4d2b-af06-87dd7beb0ebc 15:46:35.513 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:46:35.513 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment d1951373-a638-4d2b-af06-87dd7beb0ebc for client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:35.513 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment d1951373-a638-4d2b-af06-87dd7beb0ebc to client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:35.513 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 148d14a6-388c-4371-b59f-5863c4b74903 io.netty.handler.codec.compression.DecompressionException: mismatching checksum: b9950dce (expected: a281f895) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] 15:46:40.532 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 148d14a6-388c-4371-b59f-5863c4b74903 io.netty.handler.timeout.ReadTimeoutException: null 15:46:40.532 DEBUG [standby-run-37] StandbyClient.java:157 Channel closed 15:46:40.532 ERROR [standby-run-37] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment d1951373-a638-4d2b-af06-87dd7beb0ebc from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:340) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByte(DataStoreTestBase.java:304) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_60] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_60] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_60] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_60] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121) [surefire-booter-2.19.1.jar:2.19.1] 15:46:40.532 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=-1, skip=-1,0 15:46:40.532 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client 148d14a6-388c-4371-b59f-5863c4b74903 for current head 15:46:40.579 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:53629 is allowed 15:46:40.579 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 15:46:40.579 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get head' received from client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.579 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.579 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head d1951373-a638-4d2b-af06-87dd7beb0ebc.0000000c to client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.579 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 15:46:40.579 DEBUG [standby-run-38] StandbyClientSyncExecution.java:103 Found missing segment d1951373-a638-4d2b-af06-87dd7beb0ebc 15:46:40.579 DEBUG [standby-run-38] StandbyClientSyncExecution.java:124 Inspecting segment d1951373-a638-4d2b-af06-87dd7beb0ebc 15:46:40.579 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 148d14a6-388c-4371-b59f-5863c4b74903 for references of segment d1951373-a638-4d2b-af06-87dd7beb0ebc 15:46:40.579 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:46:40.579 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment d1951373-a638-4d2b-af06-87dd7beb0ebc for client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.579 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment d1951373-a638-4d2b-af06-87dd7beb0ebc to client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.579 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:46:40.579 DEBUG [standby-run-38] StandbyClientSyncExecution.java:184 Found reference from d1951373-a638-4d2b-af06-87dd7beb0ebc to 8d8b434b-28cd-4f20-ac18-319e3caf1e6e 15:46:40.579 DEBUG [standby-run-38] StandbyClientSyncExecution.java:124 Inspecting segment 8d8b434b-28cd-4f20-ac18-319e3caf1e6e 15:46:40.579 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 148d14a6-388c-4371-b59f-5863c4b74903 for references of segment 8d8b434b-28cd-4f20-ac18-319e3caf1e6e 15:46:40.579 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:46:40.579 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 8d8b434b-28cd-4f20-ac18-319e3caf1e6e for client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.579 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 8d8b434b-28cd-4f20-ac18-319e3caf1e6e to client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.579 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:46:40.579 INFO [standby-run-38] StandbyClientSyncExecution.java:196 Copying data segment 8d8b434b-28cd-4f20-ac18-319e3caf1e6e from primary 15:46:40.579 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 148d14a6-388c-4371-b59f-5863c4b74903 for segment 8d8b434b-28cd-4f20-ac18-319e3caf1e6e 15:46:40.579 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:46:40.579 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.579 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 8d8b434b-28cd-4f20-ac18-319e3caf1e6e for client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.579 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 192 sent to client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.579 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 8d8b434b-28cd-4f20-ac18-319e3caf1e6e to client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.579 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:40.579 INFO [standby-run-38] StandbyClientSyncExecution.java:196 Copying data segment d1951373-a638-4d2b-af06-87dd7beb0ebc from primary 15:46:40.579 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 148d14a6-388c-4371-b59f-5863c4b74903 for segment d1951373-a638-4d2b-af06-87dd7beb0ebc 15:46:40.579 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:46:40.579 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.579 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment d1951373-a638-4d2b-af06-87dd7beb0ebc for client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.579 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 448 sent to client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.579 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment d1951373-a638-4d2b-af06-87dd7beb0ebc to client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.579 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:40.579 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client 148d14a6-388c-4371-b59f-5863c4b74903 for blob 557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880 15:46:40.594 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 15:46:40.594 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get blob id' received from client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.594 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880 for client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.594 DEBUG [primary-2] CommunicationObserver.java:105 Binary with size 5242880 sent to client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.594 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880 to client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.594 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880 to client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.616 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880 to client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.632 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:40.632 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880 15:46:40.632 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880.tmp 15:46:40.648 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880 to client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.679 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880 to client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.732 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:40.732 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880 15:46:40.732 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880 to client 148d14a6-388c-4371-b59f-5863c4b74903 15:46:40.732 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880.tmp 15:46:40.764 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:40.764 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880 15:46:40.764 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880.tmp 15:46:40.779 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:40.779 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880 15:46:40.779 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880.tmp 15:46:40.795 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:40.795 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880 15:46:40.812 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880.tmp 15:46:40.816 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob 557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880 15:46:40.864 DEBUG [standby-run-38] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\557c9632854e08d7504657f6bcae383a2a568f7544752a86eba1edb391f677e4#5242880.tmp 15:46:40.995 INFO [standby-run-38] StandbyClientSyncExecution.java:82 updated head state successfully: true in 416ms. 15:46:40.995 DEBUG [standby-run-38] StandbyClient.java:157 Channel closed 15:46:43.335 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:46:43.335 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:46:43.335 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:46:43.335 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:46:43.335 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:46:43.382 INFO [main] FileStore.java:471 TarMK closed: target\junit4451559270981989588\junit5485841426612374562 15:46:43.398 INFO [main] FileStore.java:471 TarMK closed: target\junit4451559270981989588\junit3939919258085895365 15:46:43.436 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit6133508877836871842\junit6875078691129672596, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:46:43.436 INFO [main] FileStore.java:240 TarMK opened: target\junit6133508877836871842\junit6875078691129672596 (mmap=false) 15:46:43.436 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit6133508877836871842\junit8235407594841182404, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:46:43.436 INFO [main] FileStore.java:240 TarMK opened: target\junit6133508877836871842\junit8235407594841182404 (mmap=false) 15:46:43.520 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:46:43.520 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:46:43.620 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=150000, skip=0,0 15:46:43.636 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x4d38d972]' 15:46:43.636 DEBUG [main] StandbyServer.java:249 Binding was successful 15:46:44.203 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client c730cf65-f79a-48ac-a0c3-e25bc315c75d for current head 15:46:44.256 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:53726 is allowed 15:46:44.256 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:46:44.256 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.256 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.256 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 66e32c4f-64aa-45e2-a981-0b0b22def3d0.0000000c to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.256 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:46:44.256 DEBUG [standby-run-39] StandbyClientSyncExecution.java:103 Found missing segment 66e32c4f-64aa-45e2-a981-0b0b22def3d0 15:46:44.256 DEBUG [standby-run-39] StandbyClientSyncExecution.java:124 Inspecting segment 66e32c4f-64aa-45e2-a981-0b0b22def3d0 15:46:44.256 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client c730cf65-f79a-48ac-a0c3-e25bc315c75d for references of segment 66e32c4f-64aa-45e2-a981-0b0b22def3d0 15:46:44.256 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:46:44.256 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 66e32c4f-64aa-45e2-a981-0b0b22def3d0 for client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.256 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 66e32c4f-64aa-45e2-a981-0b0b22def3d0 to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.256 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:46:44.256 DEBUG [standby-run-39] StandbyClientSyncExecution.java:184 Found reference from 66e32c4f-64aa-45e2-a981-0b0b22def3d0 to 44bea06e-30cd-45b1-a9fc-260158b89ec7 15:46:44.256 DEBUG [standby-run-39] StandbyClientSyncExecution.java:124 Inspecting segment 44bea06e-30cd-45b1-a9fc-260158b89ec7 15:46:44.256 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client c730cf65-f79a-48ac-a0c3-e25bc315c75d for references of segment 44bea06e-30cd-45b1-a9fc-260158b89ec7 15:46:44.256 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:46:44.272 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 44bea06e-30cd-45b1-a9fc-260158b89ec7 for client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.272 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 44bea06e-30cd-45b1-a9fc-260158b89ec7 to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.272 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:46:44.272 INFO [standby-run-39] StandbyClientSyncExecution.java:196 Copying data segment 44bea06e-30cd-45b1-a9fc-260158b89ec7 from primary 15:46:44.272 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client c730cf65-f79a-48ac-a0c3-e25bc315c75d for segment 44bea06e-30cd-45b1-a9fc-260158b89ec7 15:46:44.272 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:46:44.272 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.272 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 44bea06e-30cd-45b1-a9fc-260158b89ec7 for client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.272 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 192 sent to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.272 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 44bea06e-30cd-45b1-a9fc-260158b89ec7 to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.272 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:44.272 INFO [standby-run-39] StandbyClientSyncExecution.java:196 Copying data segment 66e32c4f-64aa-45e2-a981-0b0b22def3d0 from primary 15:46:44.272 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client c730cf65-f79a-48ac-a0c3-e25bc315c75d for segment 66e32c4f-64aa-45e2-a981-0b0b22def3d0 15:46:44.272 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:46:44.272 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.272 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 66e32c4f-64aa-45e2-a981-0b0b22def3d0 for client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.272 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 448 sent to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.272 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 66e32c4f-64aa-45e2-a981-0b0b22def3d0 to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.272 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:44.272 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client c730cf65-f79a-48ac-a0c3-e25bc315c75d for blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 15:46:44.272 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 15:46:44.272 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get blob id' received from client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.272 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 for client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.272 DEBUG [primary-1] CommunicationObserver.java:105 Binary with size 5242880 sent to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.272 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.272 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.288 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client c730cf65-f79a-48ac-a0c3-e25bc315c75d io.netty.handler.codec.compression.DecompressionException: mismatching checksum: 9d97df0d (expected: b4055a00) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] 15:46:44.303 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.341 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.357 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:44.372 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:49.278 DEBUG [standby-run-39] StandbyClient.java:157 Channel closed 15:46:49.278 ERROR [standby-run-39] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to load remote blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 at /root/server/#testBlob in 5000ms. Please increase the timeout and try again. at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchAndStoreBlob(StandbyDiff.java:208) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:196) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:181) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:169) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeAdded(StandbyDiff.java:94) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeAdded(CancelableDiff.java:76) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:562) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:566) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:101) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:340) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByte2(DataStoreTestBase.java:309) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_60] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_60] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_60] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_60] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121) [surefire-booter-2.19.1.jar:2.19.1] 15:46:49.278 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=-1, skip=-1,0 15:46:49.293 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client c730cf65-f79a-48ac-a0c3-e25bc315c75d for current head 15:46:49.331 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:53779 is allowed 15:46:49.331 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 15:46:49.331 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get head' received from client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:49.331 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:49.331 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 66e32c4f-64aa-45e2-a981-0b0b22def3d0.0000000c to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:49.331 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 15:46:49.331 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client c730cf65-f79a-48ac-a0c3-e25bc315c75d for blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 15:46:49.331 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 15:46:49.331 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get blob id' received from client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:49.331 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 for client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:49.331 DEBUG [primary-2] CommunicationObserver.java:105 Binary with size 5242880 sent to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:49.331 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:49.331 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:49.362 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:49.378 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:49.378 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 15:46:49.378 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880.tmp 15:46:49.394 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:49.409 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:49.409 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 15:46:49.431 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:49.431 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880.tmp 15:46:49.463 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 to client c730cf65-f79a-48ac-a0c3-e25bc315c75d 15:46:49.494 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:49.494 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 15:46:49.494 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880.tmp 15:46:49.509 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:49.509 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 15:46:49.509 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880.tmp 15:46:49.532 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:49.532 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 15:46:49.532 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880.tmp 15:46:49.532 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob 170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880 15:46:49.594 DEBUG [standby-run-40] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\170facc5d491dbb9e81ab207f562782aec690d9466bce2470d048eb3e702f1bf#5242880.tmp 15:46:49.694 INFO [standby-run-40] StandbyClientSyncExecution.java:82 updated head state successfully: true in 363ms. 15:46:49.694 DEBUG [standby-run-40] StandbyClient.java:157 Channel closed 15:46:52.048 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:46:52.048 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:46:52.048 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:46:52.048 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:46:52.048 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:46:52.079 INFO [main] FileStore.java:471 TarMK closed: target\junit6133508877836871842\junit8235407594841182404 15:46:52.110 INFO [main] FileStore.java:471 TarMK closed: target\junit6133508877836871842\junit6875078691129672596 15:46:52.132 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1639598158726861179\junit3437155850395219423, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:46:52.132 INFO [main] FileStore.java:240 TarMK opened: target\junit1639598158726861179\junit3437155850395219423 (mmap=false) 15:46:52.148 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1639598158726861179\junit7352501282063415918, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:46:52.148 INFO [main] FileStore.java:240 TarMK opened: target\junit1639598158726861179\junit7352501282063415918 (mmap=false) 15:46:52.195 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:46:52.195 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:46:52.295 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=0, skip=0,0 15:46:52.333 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x495653e8]' 15:46:52.333 DEBUG [main] StandbyServer.java:249 Binding was successful 15:46:52.464 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client f0155d9d-a9ea-425e-a054-f4171094df3a for current head 15:46:52.495 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:53879 is allowed 15:46:52.495 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:46:52.495 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:52.495 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:52.495 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 298aa76a-42d4-4202-a5d5-350ada970fdb.0000000c to client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:52.495 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client f0155d9d-a9ea-425e-a054-f4171094df3a io.netty.handler.codec.compression.DecompressionException: Received COMPRESSED_DATA tag before STREAM_IDENTIFIER at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:162) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] 15:46:57.488 ERROR [standby-run-41] StandbyClientSyncExecution.java:67 Unable to fetch remote head 15:46:57.488 DEBUG [standby-run-41] StandbyClient.java:157 Channel closed 15:46:57.488 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=-1, skip=-1,0 15:46:57.504 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client f0155d9d-a9ea-425e-a054-f4171094df3a for current head 15:46:57.541 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:53904 is allowed 15:46:57.541 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 15:46:57.541 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get head' received from client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.541 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.541 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 298aa76a-42d4-4202-a5d5-350ada970fdb.0000000c to client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.541 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 15:46:57.541 DEBUG [standby-run-42] StandbyClientSyncExecution.java:103 Found missing segment 298aa76a-42d4-4202-a5d5-350ada970fdb 15:46:57.541 DEBUG [standby-run-42] StandbyClientSyncExecution.java:124 Inspecting segment 298aa76a-42d4-4202-a5d5-350ada970fdb 15:46:57.541 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client f0155d9d-a9ea-425e-a054-f4171094df3a for references of segment 298aa76a-42d4-4202-a5d5-350ada970fdb 15:46:57.541 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:46:57.541 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 298aa76a-42d4-4202-a5d5-350ada970fdb for client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.541 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 298aa76a-42d4-4202-a5d5-350ada970fdb to client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.541 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:46:57.541 DEBUG [standby-run-42] StandbyClientSyncExecution.java:184 Found reference from 298aa76a-42d4-4202-a5d5-350ada970fdb to 03369ecd-7e01-467b-af89-8dc7127db33a 15:46:57.541 DEBUG [standby-run-42] StandbyClientSyncExecution.java:124 Inspecting segment 03369ecd-7e01-467b-af89-8dc7127db33a 15:46:57.541 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client f0155d9d-a9ea-425e-a054-f4171094df3a for references of segment 03369ecd-7e01-467b-af89-8dc7127db33a 15:46:57.541 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:46:57.541 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 03369ecd-7e01-467b-af89-8dc7127db33a for client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.541 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 03369ecd-7e01-467b-af89-8dc7127db33a to client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.541 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:46:57.541 INFO [standby-run-42] StandbyClientSyncExecution.java:196 Copying data segment 03369ecd-7e01-467b-af89-8dc7127db33a from primary 15:46:57.541 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client f0155d9d-a9ea-425e-a054-f4171094df3a for segment 03369ecd-7e01-467b-af89-8dc7127db33a 15:46:57.541 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:46:57.541 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.541 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 03369ecd-7e01-467b-af89-8dc7127db33a for client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.541 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 192 sent to client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.541 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 03369ecd-7e01-467b-af89-8dc7127db33a to client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.541 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:57.541 INFO [standby-run-42] StandbyClientSyncExecution.java:196 Copying data segment 298aa76a-42d4-4202-a5d5-350ada970fdb from primary 15:46:57.541 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client f0155d9d-a9ea-425e-a054-f4171094df3a for segment 298aa76a-42d4-4202-a5d5-350ada970fdb 15:46:57.541 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:46:57.541 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.541 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 298aa76a-42d4-4202-a5d5-350ada970fdb for client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.541 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 448 sent to client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.541 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 298aa76a-42d4-4202-a5d5-350ada970fdb to client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.557 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:46:57.557 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client f0155d9d-a9ea-425e-a054-f4171094df3a for blob b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880 15:46:57.557 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 15:46:57.557 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get blob id' received from client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.557 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880 for client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.557 DEBUG [primary-2] CommunicationObserver.java:105 Binary with size 5242880 sent to client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.557 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880 to client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.557 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880 to client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.573 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880 to client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.588 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:57.588 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880 15:46:57.588 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880.tmp 15:46:57.604 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880 to client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.619 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880 to client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.619 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:57.619 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880 15:46:57.619 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880.tmp 15:46:57.642 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880 to client f0155d9d-a9ea-425e-a054-f4171094df3a 15:46:57.642 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:57.642 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880 15:46:57.657 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880.tmp 15:46:57.673 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:57.673 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880 15:46:57.673 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880.tmp 15:46:57.689 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:46:57.689 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880 15:46:57.689 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880.tmp 15:46:57.704 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880 15:46:57.742 DEBUG [standby-run-42] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\b1619d72761059f2b37e1c36f7146f332c83b61705ea7a05ba243ef3148330c2#5242880.tmp 15:46:57.888 INFO [standby-run-42] StandbyClientSyncExecution.java:82 updated head state successfully: true in 347ms. 15:46:57.888 DEBUG [standby-run-42] StandbyClient.java:157 Channel closed 15:47:00.221 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:47:00.221 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:47:00.221 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:47:00.221 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:47:00.221 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:47:00.259 INFO [main] FileStore.java:471 TarMK closed: target\junit1639598158726861179\junit7352501282063415918 15:47:00.291 INFO [main] FileStore.java:471 TarMK closed: target\junit1639598158726861179\junit3437155850395219423 15:47:00.322 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit2030577273074257876\junit218421847710381770, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:47:00.322 INFO [main] FileStore.java:240 TarMK opened: target\junit2030577273074257876\junit218421847710381770 (mmap=false) 15:47:00.338 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit2030577273074257876\junit5468772671357673418, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:47:00.340 INFO [main] FileStore.java:240 TarMK opened: target\junit2030577273074257876\junit5468772671357673418 (mmap=false) 15:47:00.422 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:47:00.422 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:47:29.277 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x1a0f3c69]' 15:47:29.277 DEBUG [main] StandbyServer.java:249 Binding was successful 15:47:29.293 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client d5a23d77-f065-4ee3-b123-2e5c111f1e59 for current head 15:47:29.293 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:54015 is allowed 15:47:29.293 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:47:29.293 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client d5a23d77-f065-4ee3-b123-2e5c111f1e59 15:47:29.293 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client d5a23d77-f065-4ee3-b123-2e5c111f1e59 15:47:29.293 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 5cd61aa0-8400-4201-a608-68722bf8ceb2.00000005 to client d5a23d77-f065-4ee3-b123-2e5c111f1e59 15:47:29.293 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:47:29.293 DEBUG [standby-run-43] StandbyClientSyncExecution.java:103 Found missing segment 5cd61aa0-8400-4201-a608-68722bf8ceb2 15:47:29.293 DEBUG [standby-run-43] StandbyClientSyncExecution.java:124 Inspecting segment 5cd61aa0-8400-4201-a608-68722bf8ceb2 15:47:29.293 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client d5a23d77-f065-4ee3-b123-2e5c111f1e59 for references of segment 5cd61aa0-8400-4201-a608-68722bf8ceb2 15:47:29.293 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:47:29.293 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 5cd61aa0-8400-4201-a608-68722bf8ceb2 for client d5a23d77-f065-4ee3-b123-2e5c111f1e59 15:47:29.393 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 5cd61aa0-8400-4201-a608-68722bf8ceb2 to client d5a23d77-f065-4ee3-b123-2e5c111f1e59 15:47:29.393 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:47:29.393 INFO [standby-run-43] StandbyClientSyncExecution.java:196 Copying data segment 5cd61aa0-8400-4201-a608-68722bf8ceb2 from primary 15:47:29.408 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client d5a23d77-f065-4ee3-b123-2e5c111f1e59 for segment 5cd61aa0-8400-4201-a608-68722bf8ceb2 15:47:29.408 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:47:29.408 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client d5a23d77-f065-4ee3-b123-2e5c111f1e59 15:47:29.408 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 5cd61aa0-8400-4201-a608-68722bf8ceb2 for client d5a23d77-f065-4ee3-b123-2e5c111f1e59 15:47:29.408 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 192 sent to client d5a23d77-f065-4ee3-b123-2e5c111f1e59 15:47:29.408 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 5cd61aa0-8400-4201-a608-68722bf8ceb2 to client d5a23d77-f065-4ee3-b123-2e5c111f1e59 15:47:29.408 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:47:29.758 INFO [standby-run-43] StandbyClientSyncExecution.java:82 updated head state successfully: true in 465ms. 15:47:29.759 DEBUG [standby-run-43] StandbyClient.java:157 Channel closed 15:47:31.994 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:47:31.994 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:47:31.994 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:47:31.994 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:47:31.994 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:47:32.063 INFO [main] FileStore.java:471 TarMK closed: target\junit2030577273074257876\junit5468772671357673418 15:47:32.094 INFO [main] FileStore.java:471 TarMK closed: target\junit2030577273074257876\junit218421847710381770