15:01:24.878 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit2499293097700684604\junit6972956420706447257, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:01:24.878 INFO [main] FileStore.java:240 TarMK opened: target\junit2499293097700684604\junit6972956420706447257 (mmap=false) 15:01:24.878 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit2499293097700684604\junit2818032985351306173, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:01:24.897 INFO [main] FileStore.java:240 TarMK opened: target\junit2499293097700684604\junit2818032985351306173 (mmap=false) 15:01:24.947 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:01:24.947 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:01:25.063 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x55445765]' 15:01:25.063 DEBUG [main] StandbyServer.java:249 Binding was successful 15:01:25.216 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client a72ae904-7549-4384-87e2-34fe87295281 for current head 15:01:25.216 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:58392 is allowed 15:01:25.232 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:01:25.232 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.232 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.232 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 3265a626-733d-4cd7-a1ce-89d0bd17121b.0000000c to client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.232 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:01:25.232 DEBUG [standby-run-23] StandbyClientSyncExecution.java:103 Found missing segment 3265a626-733d-4cd7-a1ce-89d0bd17121b 15:01:25.232 DEBUG [standby-run-23] StandbyClientSyncExecution.java:124 Inspecting segment 3265a626-733d-4cd7-a1ce-89d0bd17121b 15:01:25.232 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client a72ae904-7549-4384-87e2-34fe87295281 for references of segment 3265a626-733d-4cd7-a1ce-89d0bd17121b 15:01:25.232 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:01:25.232 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 3265a626-733d-4cd7-a1ce-89d0bd17121b for client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.232 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 3265a626-733d-4cd7-a1ce-89d0bd17121b to client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.232 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:01:25.232 DEBUG [standby-run-23] StandbyClientSyncExecution.java:184 Found reference from 3265a626-733d-4cd7-a1ce-89d0bd17121b to a0efe79e-1560-4577-a2e7-dcb8275d3075 15:01:25.232 DEBUG [standby-run-23] StandbyClientSyncExecution.java:124 Inspecting segment a0efe79e-1560-4577-a2e7-dcb8275d3075 15:01:25.232 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client a72ae904-7549-4384-87e2-34fe87295281 for references of segment a0efe79e-1560-4577-a2e7-dcb8275d3075 15:01:25.232 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:01:25.232 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment a0efe79e-1560-4577-a2e7-dcb8275d3075 for client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.232 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment a0efe79e-1560-4577-a2e7-dcb8275d3075 to client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.232 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:01:25.232 INFO [standby-run-23] StandbyClientSyncExecution.java:196 Copying data segment a0efe79e-1560-4577-a2e7-dcb8275d3075 from primary 15:01:25.232 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client a72ae904-7549-4384-87e2-34fe87295281 for segment a0efe79e-1560-4577-a2e7-dcb8275d3075 15:01:25.247 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:01:25.247 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.247 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment a0efe79e-1560-4577-a2e7-dcb8275d3075 for client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.247 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 192 sent to client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.247 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment a0efe79e-1560-4577-a2e7-dcb8275d3075 to client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.247 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:01:25.247 INFO [standby-run-23] StandbyClientSyncExecution.java:196 Copying data segment 3265a626-733d-4cd7-a1ce-89d0bd17121b from primary 15:01:25.247 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client a72ae904-7549-4384-87e2-34fe87295281 for segment 3265a626-733d-4cd7-a1ce-89d0bd17121b 15:01:25.247 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:01:25.247 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.247 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 3265a626-733d-4cd7-a1ce-89d0bd17121b for client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.247 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 448 sent to client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.247 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 3265a626-733d-4cd7-a1ce-89d0bd17121b to client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.247 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:01:25.247 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client a72ae904-7549-4384-87e2-34fe87295281 for blob 49c3d60c685fdd5f6ca939c8b5dabc54d345154f09659e823133ec5ed4b1853d#5242880 15:01:25.263 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 15:01:25.263 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get blob id' received from client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.263 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob 49c3d60c685fdd5f6ca939c8b5dabc54d345154f09659e823133ec5ed4b1853d#5242880 for client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.263 DEBUG [primary-1] CommunicationObserver.java:105 Binary with size 5242880 sent to client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.263 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob 49c3d60c685fdd5f6ca939c8b5dabc54d345154f09659e823133ec5ed4b1853d#5242880 to client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.301 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 49c3d60c685fdd5f6ca939c8b5dabc54d345154f09659e823133ec5ed4b1853d#5242880 to client a72ae904-7549-4384-87e2-34fe87295281 15:01:25.332 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client a72ae904-7549-4384-87e2-34fe87295281 io.netty.handler.timeout.ReadTimeoutException: null 15:01:25.332 DEBUG [standby-run-23] StandbyClient.java:157 Channel closed 15:01:25.332 ERROR [standby-run-23] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to load remote blob 49c3d60c685fdd5f6ca939c8b5dabc54d345154f09659e823133ec5ed4b1853d#5242880 at /root/server/#testBlob in 60ms. Please increase the timeout and try again. at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchAndStoreBlob(StandbyDiff.java:208) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:196) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:181) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:169) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeAdded(StandbyDiff.java:94) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeAdded(CancelableDiff.java:76) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:562) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:566) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:101) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.ExternalPrivateStoreIT.testSyncFailingDueToTooShortTimeout(ExternalPrivateStoreIT.java:86) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_60] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_60] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_60] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_60] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121) [surefire-booter-2.19.1.jar:2.19.1] 15:01:27.563 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:01:27.563 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:01:27.563 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:01:27.563 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:01:27.563 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:01:27.816 INFO [main] FileStore.java:471 TarMK closed: target\junit2499293097700684604\junit2818032985351306173 15:01:27.863 INFO [main] FileStore.java:471 TarMK closed: target\junit2499293097700684604\junit6972956420706447257 15:01:27.879 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7496927196359077293\junit7937203383382017923, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:01:27.879 INFO [main] FileStore.java:240 TarMK opened: target\junit7496927196359077293\junit7937203383382017923 (mmap=false) 15:01:27.879 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7496927196359077293\junit6806349163708527870, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:01:27.879 INFO [main] FileStore.java:240 TarMK opened: target\junit7496927196359077293\junit6806349163708527870 (mmap=false) 15:01:27.932 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:01:27.932 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:01:28.032 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x06c459a3]' 15:01:28.048 DEBUG [main] StandbyServer.java:249 Binding was successful 15:01:28.048 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 06d1a5c4-83b6-4ff0-a3ae-fcbf53041270 for current head 15:01:28.048 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:58462 is allowed 15:01:28.048 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:01:28.048 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client 06d1a5c4-83b6-4ff0-a3ae-fcbf53041270 15:01:28.048 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 06d1a5c4-83b6-4ff0-a3ae-fcbf53041270 15:01:32.052 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 06d1a5c4-83b6-4ff0-a3ae-fcbf53041270 io.netty.handler.timeout.ReadTimeoutException: null 15:01:32.053 ERROR [standby-run-24] StandbyClientSyncExecution.java:67 Unable to fetch remote head 15:01:32.054 DEBUG [standby-run-24] StandbyClient.java:157 Channel closed 15:01:33.203 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head b622b81a-421d-46b3-a82b-768ccedd78e6.0000000c to client 06d1a5c4-83b6-4ff0-a3ae-fcbf53041270 15:01:33.203 WARN [primary-1] ExceptionHandler.java:31 Exception caught on the server java.io.IOException: Eine bestehende Verbindung wurde softwaregesteuert durch den Hostcomputer abgebrochen at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[na:1.8.0_60] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) ~[na:1.8.0_60] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_60] at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_60] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_60] at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288) ~[netty-buffer-4.1.14.Final.jar:4.1.14.Final] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106) ~[netty-buffer-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:372) ~[netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) ~[netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] 15:01:34.259 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:01:34.259 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:01:34.259 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:01:34.259 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:01:34.259 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:01:34.323 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x16a43ffe]' 15:01:34.325 DEBUG [main] StandbyServer.java:249 Binding was successful 15:01:34.328 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f for current head 15:01:34.328 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:58498 is allowed 15:01:34.328 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:01:34.328 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.328 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.328 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head b622b81a-421d-46b3-a82b-768ccedd78e6.0000000c to client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.328 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:01:34.328 DEBUG [standby-run-25] StandbyClientSyncExecution.java:103 Found missing segment b622b81a-421d-46b3-a82b-768ccedd78e6 15:01:34.328 DEBUG [standby-run-25] StandbyClientSyncExecution.java:124 Inspecting segment b622b81a-421d-46b3-a82b-768ccedd78e6 15:01:34.328 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f for references of segment b622b81a-421d-46b3-a82b-768ccedd78e6 15:01:34.328 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:01:34.328 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment b622b81a-421d-46b3-a82b-768ccedd78e6 for client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.328 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment b622b81a-421d-46b3-a82b-768ccedd78e6 to client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.328 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:01:34.328 DEBUG [standby-run-25] StandbyClientSyncExecution.java:184 Found reference from b622b81a-421d-46b3-a82b-768ccedd78e6 to f59d6bbf-88d0-4a89-a84b-bc4add733eef 15:01:34.328 DEBUG [standby-run-25] StandbyClientSyncExecution.java:124 Inspecting segment f59d6bbf-88d0-4a89-a84b-bc4add733eef 15:01:34.328 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f for references of segment f59d6bbf-88d0-4a89-a84b-bc4add733eef 15:01:34.328 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:01:34.328 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment f59d6bbf-88d0-4a89-a84b-bc4add733eef for client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.328 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment f59d6bbf-88d0-4a89-a84b-bc4add733eef to client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.328 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:01:34.328 INFO [standby-run-25] StandbyClientSyncExecution.java:196 Copying data segment f59d6bbf-88d0-4a89-a84b-bc4add733eef from primary 15:01:34.328 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f for segment f59d6bbf-88d0-4a89-a84b-bc4add733eef 15:01:34.328 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:01:34.328 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.328 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment f59d6bbf-88d0-4a89-a84b-bc4add733eef for client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.328 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 192 sent to client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.328 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment f59d6bbf-88d0-4a89-a84b-bc4add733eef to client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.328 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:01:34.328 INFO [standby-run-25] StandbyClientSyncExecution.java:196 Copying data segment b622b81a-421d-46b3-a82b-768ccedd78e6 from primary 15:01:34.328 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f for segment b622b81a-421d-46b3-a82b-768ccedd78e6 15:01:34.328 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:01:34.328 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.328 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment b622b81a-421d-46b3-a82b-768ccedd78e6 for client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.328 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 448 sent to client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.328 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment b622b81a-421d-46b3-a82b-768ccedd78e6 to client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.328 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:01:34.328 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f for blob 3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880 15:01:34.328 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 15:01:34.328 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get blob id' received from client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.328 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob 3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880 for client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.328 DEBUG [primary-1] CommunicationObserver.java:105 Binary with size 5242880 sent to client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.328 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob 3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880 to client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.344 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880 to client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.360 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880 to client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.360 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:34.360 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880 15:01:34.360 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880.tmp 15:01:34.375 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880 to client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.391 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:34.391 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880 15:01:34.391 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880.tmp 15:01:34.391 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880 to client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.406 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:34.406 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880 15:01:34.406 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880.tmp 15:01:34.406 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880 to client 4b4d1870-2297-4692-a00a-d9f5c4f9eb4f 15:01:34.429 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:34.429 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880 15:01:34.429 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880.tmp 15:01:34.429 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:34.429 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880 15:01:34.444 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880.tmp 15:01:34.444 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob 3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880 15:01:34.491 DEBUG [standby-run-25] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\3ebd3db9df58d51d8142a2fde16acc64ba35f30f6df39fc4e5fbb40f76b0f61d#5242880.tmp 15:01:34.660 INFO [standby-run-25] StandbyClientSyncExecution.java:82 updated head state successfully: true in 332ms. 15:01:34.660 DEBUG [standby-run-25] StandbyClient.java:157 Channel closed 15:01:36.861 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:01:36.861 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:01:36.861 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:01:36.861 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:01:36.861 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:01:36.961 INFO [main] FileStore.java:471 TarMK closed: target\junit7496927196359077293\junit6806349163708527870 15:01:37.008 INFO [main] FileStore.java:471 TarMK closed: target\junit7496927196359077293\junit7937203383382017923 15:01:37.030 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit496762422529115419\junit1809841927983018362, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:01:37.030 INFO [main] FileStore.java:240 TarMK opened: target\junit496762422529115419\junit1809841927983018362 (mmap=false) 15:01:37.077 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit496762422529115419\junit6282172199668030275, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:01:37.077 INFO [main] FileStore.java:240 TarMK opened: target\junit496762422529115419\junit6282172199668030275 (mmap=false) 15:01:37.162 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:01:37.162 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:01:37.246 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=-1, skip=100,1 15:01:37.278 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xedeef596]' 15:01:37.278 DEBUG [main] StandbyServer.java:249 Binding was successful 15:01:37.631 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 36c361e2-fd7a-45ac-b5f3-455fcf78583b for current head 15:01:37.662 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:58586 is allowed 15:01:37.662 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:01:37.662 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:37.662 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:37.662 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 4b620d14-e290-4690-aeef-44affe73aa5c.0000000c to client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:37.662 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:01:37.662 DEBUG [standby-run-26] StandbyClientSyncExecution.java:103 Found missing segment 4b620d14-e290-4690-aeef-44affe73aa5c 15:01:37.662 DEBUG [standby-run-26] StandbyClientSyncExecution.java:124 Inspecting segment 4b620d14-e290-4690-aeef-44affe73aa5c 15:01:37.662 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 36c361e2-fd7a-45ac-b5f3-455fcf78583b for references of segment 4b620d14-e290-4690-aeef-44affe73aa5c 15:01:37.662 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:01:37.662 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 4b620d14-e290-4690-aeef-44affe73aa5c for client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:37.662 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 4b620d14-e290-4690-aeef-44affe73aa5c to client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.681 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 36c361e2-fd7a-45ac-b5f3-455fcf78583b io.netty.handler.timeout.ReadTimeoutException: null 15:01:42.681 DEBUG [standby-run-26] StandbyClient.java:157 Channel closed 15:01:42.681 ERROR [standby-run-26] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment 4b620d14-e290-4690-aeef-44affe73aa5c from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:340) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxySkippedBytes(DataStoreTestBase.java:289) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_60] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_60] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_60] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_60] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121) [surefire-booter-2.19.1.jar:2.19.1] 15:01:42.681 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=-1, skip=-1,0 15:01:42.696 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client 36c361e2-fd7a-45ac-b5f3-455fcf78583b for current head 15:01:42.733 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:58604 is allowed 15:01:42.733 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 15:01:42.733 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get head' received from client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.733 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.733 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 4b620d14-e290-4690-aeef-44affe73aa5c.0000000c to client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.733 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 15:01:42.733 DEBUG [standby-run-27] StandbyClientSyncExecution.java:103 Found missing segment 4b620d14-e290-4690-aeef-44affe73aa5c 15:01:42.733 DEBUG [standby-run-27] StandbyClientSyncExecution.java:124 Inspecting segment 4b620d14-e290-4690-aeef-44affe73aa5c 15:01:42.733 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 36c361e2-fd7a-45ac-b5f3-455fcf78583b for references of segment 4b620d14-e290-4690-aeef-44affe73aa5c 15:01:42.733 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:01:42.733 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 4b620d14-e290-4690-aeef-44affe73aa5c for client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.733 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 4b620d14-e290-4690-aeef-44affe73aa5c to client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.750 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:01:42.750 DEBUG [standby-run-27] StandbyClientSyncExecution.java:184 Found reference from 4b620d14-e290-4690-aeef-44affe73aa5c to 4e55af80-1fb7-4813-a950-fe1d61872800 15:01:42.750 DEBUG [standby-run-27] StandbyClientSyncExecution.java:124 Inspecting segment 4e55af80-1fb7-4813-a950-fe1d61872800 15:01:42.750 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 36c361e2-fd7a-45ac-b5f3-455fcf78583b for references of segment 4e55af80-1fb7-4813-a950-fe1d61872800 15:01:42.750 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:01:42.750 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 4e55af80-1fb7-4813-a950-fe1d61872800 for client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.750 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 4e55af80-1fb7-4813-a950-fe1d61872800 to client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.750 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:01:42.750 INFO [standby-run-27] StandbyClientSyncExecution.java:196 Copying data segment 4e55af80-1fb7-4813-a950-fe1d61872800 from primary 15:01:42.750 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 36c361e2-fd7a-45ac-b5f3-455fcf78583b for segment 4e55af80-1fb7-4813-a950-fe1d61872800 15:01:42.750 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:01:42.750 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.750 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 4e55af80-1fb7-4813-a950-fe1d61872800 for client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.750 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 192 sent to client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.750 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 4e55af80-1fb7-4813-a950-fe1d61872800 to client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.750 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:01:42.750 INFO [standby-run-27] StandbyClientSyncExecution.java:196 Copying data segment 4b620d14-e290-4690-aeef-44affe73aa5c from primary 15:01:42.750 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 36c361e2-fd7a-45ac-b5f3-455fcf78583b for segment 4b620d14-e290-4690-aeef-44affe73aa5c 15:01:42.750 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:01:42.750 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.750 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 4b620d14-e290-4690-aeef-44affe73aa5c for client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.750 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 448 sent to client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.750 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 4b620d14-e290-4690-aeef-44affe73aa5c to client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.750 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:01:42.750 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client 36c361e2-fd7a-45ac-b5f3-455fcf78583b for blob 1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880 15:01:42.750 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 15:01:42.750 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get blob id' received from client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.750 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880 for client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.750 DEBUG [primary-2] CommunicationObserver.java:105 Binary with size 5242880 sent to client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.750 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880 to client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.765 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880 to client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.781 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880 to client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.796 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:42.796 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880 15:01:42.796 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880.tmp 15:01:42.796 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880 to client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.812 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:42.812 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880 15:01:42.812 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880.tmp 15:01:42.829 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880 to client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.834 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:42.834 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880 15:01:42.834 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880 to client 36c361e2-fd7a-45ac-b5f3-455fcf78583b 15:01:42.834 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880.tmp 15:01:42.850 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:42.850 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880 15:01:42.850 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880.tmp 15:01:42.865 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:42.865 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880 15:01:42.865 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880.tmp 15:01:42.865 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob 1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880 15:01:42.912 DEBUG [standby-run-27] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\1307079027ab3f927901df44f7950df1da8bd272f8454c3e21aa0d8d01da41e0#5242880.tmp 15:01:43.112 INFO [standby-run-27] StandbyClientSyncExecution.java:82 updated head state successfully: true in 379ms. 15:01:43.112 DEBUG [standby-run-27] StandbyClient.java:157 Channel closed 15:01:45.365 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:01:45.365 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:01:45.365 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:01:45.365 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:01:45.365 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:01:45.434 INFO [main] FileStore.java:471 TarMK closed: target\junit496762422529115419\junit6282172199668030275 15:01:45.497 INFO [main] FileStore.java:471 TarMK closed: target\junit496762422529115419\junit1809841927983018362 15:01:45.512 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit8409575798984762595\junit7479870917396763324, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:01:45.528 INFO [main] FileStore.java:240 TarMK opened: target\junit8409575798984762595\junit7479870917396763324 (mmap=false) 15:01:45.534 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit8409575798984762595\junit4975784373500179688, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:01:45.534 INFO [main] FileStore.java:240 TarMK opened: target\junit8409575798984762595\junit4975784373500179688 (mmap=false) 15:01:45.581 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:01:45.581 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:01:45.631 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xb43384c2]' 15:01:45.633 DEBUG [main] StandbyServer.java:249 Binding was successful 15:01:46.034 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b for current head 15:01:46.034 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:58673 is allowed 15:01:46.034 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:01:46.034 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.034 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.034 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head b03574b9-aa42-4a04-a4b4-279e10e3f521.0000000c to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.034 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:01:46.034 DEBUG [standby-run-28] StandbyClientSyncExecution.java:103 Found missing segment b03574b9-aa42-4a04-a4b4-279e10e3f521 15:01:46.034 DEBUG [standby-run-28] StandbyClientSyncExecution.java:124 Inspecting segment b03574b9-aa42-4a04-a4b4-279e10e3f521 15:01:46.034 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b for references of segment b03574b9-aa42-4a04-a4b4-279e10e3f521 15:01:46.034 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:01:46.034 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment b03574b9-aa42-4a04-a4b4-279e10e3f521 for client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.034 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment b03574b9-aa42-4a04-a4b4-279e10e3f521 to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.034 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:01:46.034 DEBUG [standby-run-28] StandbyClientSyncExecution.java:184 Found reference from b03574b9-aa42-4a04-a4b4-279e10e3f521 to 8ec73433-a4c0-4b58-a1a2-a93180fd1b9d 15:01:46.034 DEBUG [standby-run-28] StandbyClientSyncExecution.java:124 Inspecting segment 8ec73433-a4c0-4b58-a1a2-a93180fd1b9d 15:01:46.034 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b for references of segment 8ec73433-a4c0-4b58-a1a2-a93180fd1b9d 15:01:46.034 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:01:46.034 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 8ec73433-a4c0-4b58-a1a2-a93180fd1b9d for client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.034 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 8ec73433-a4c0-4b58-a1a2-a93180fd1b9d to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.034 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:01:46.034 INFO [standby-run-28] StandbyClientSyncExecution.java:196 Copying data segment 8ec73433-a4c0-4b58-a1a2-a93180fd1b9d from primary 15:01:46.034 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b for segment 8ec73433-a4c0-4b58-a1a2-a93180fd1b9d 15:01:46.034 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:01:46.034 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.034 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 8ec73433-a4c0-4b58-a1a2-a93180fd1b9d for client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.034 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 192 sent to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.034 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 8ec73433-a4c0-4b58-a1a2-a93180fd1b9d to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.034 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:01:46.034 INFO [standby-run-28] StandbyClientSyncExecution.java:196 Copying data segment b03574b9-aa42-4a04-a4b4-279e10e3f521 from primary 15:01:46.034 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b for segment b03574b9-aa42-4a04-a4b4-279e10e3f521 15:01:46.034 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:01:46.034 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.034 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment b03574b9-aa42-4a04-a4b4-279e10e3f521 for client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.034 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 448 sent to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.034 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment b03574b9-aa42-4a04-a4b4-279e10e3f521 to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.050 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:01:46.050 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b for blob 97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880 15:01:46.050 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 15:01:46.050 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get blob id' received from client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.050 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob 97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880 for client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.050 DEBUG [primary-1] CommunicationObserver.java:105 Binary with size 5242880 sent to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.050 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob 97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880 to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.050 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880 to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.065 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880 to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.065 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:46.065 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880 15:01:46.065 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880.tmp 15:01:46.081 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880 to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.081 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:46.081 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880 15:01:46.081 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880.tmp 15:01:46.097 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880 to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.097 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:46.097 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880 15:01:46.112 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880.tmp 15:01:46.112 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880 to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.128 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:46.128 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880 15:01:46.131 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880.tmp 15:01:46.135 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:46.135 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880 15:01:46.135 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880.tmp 15:01:46.151 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob 97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880 15:01:46.213 DEBUG [standby-run-28] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\97962422564c7f897ecefc0e561a37e6fd95ef294cdf130388950e6f5db845db#5242880.tmp 15:01:46.398 INFO [standby-run-28] StandbyClientSyncExecution.java:82 updated head state successfully: true in 364ms. 15:01:46.398 DEBUG [standby-run-28] StandbyClient.java:157 Channel closed 15:01:46.751 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b for current head 15:01:46.751 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:58674 is allowed 15:01:46.766 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 15:01:46.766 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get head' received from client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.766 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.766 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 77ba80bb-7a55-4ba6-ab7d-7adf5ada6e23.00000006 to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.766 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 15:01:46.766 DEBUG [standby-run-29] StandbyClientSyncExecution.java:103 Found missing segment 77ba80bb-7a55-4ba6-ab7d-7adf5ada6e23 15:01:46.766 DEBUG [standby-run-29] StandbyClientSyncExecution.java:124 Inspecting segment 77ba80bb-7a55-4ba6-ab7d-7adf5ada6e23 15:01:46.766 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b for references of segment 77ba80bb-7a55-4ba6-ab7d-7adf5ada6e23 15:01:46.766 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:01:46.766 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 77ba80bb-7a55-4ba6-ab7d-7adf5ada6e23 for client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.766 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 77ba80bb-7a55-4ba6-ab7d-7adf5ada6e23 to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.766 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:01:46.766 INFO [standby-run-29] StandbyClientSyncExecution.java:196 Copying data segment 77ba80bb-7a55-4ba6-ab7d-7adf5ada6e23 from primary 15:01:46.766 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b for segment 77ba80bb-7a55-4ba6-ab7d-7adf5ada6e23 15:01:46.766 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:01:46.766 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.766 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 77ba80bb-7a55-4ba6-ab7d-7adf5ada6e23 for client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.766 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 352 sent to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.766 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 77ba80bb-7a55-4ba6-ab7d-7adf5ada6e23 to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.766 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:01:46.766 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b for blob 19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880 15:01:46.766 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 15:01:46.766 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get blob id' received from client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.766 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880 for client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.782 DEBUG [primary-2] CommunicationObserver.java:105 Binary with size 5242880 sent to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.782 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880 to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.782 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880 to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.813 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880 to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.813 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:46.813 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880 15:01:46.813 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880.tmp 15:01:46.835 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880 to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.835 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:46.835 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880 15:01:46.835 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880.tmp 15:01:46.835 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880 to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.851 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:46.851 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880 15:01:46.851 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880.tmp 15:01:46.851 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880 to client 0ea9fd34-d01c-4c06-bf7e-42b3d40f3d9b 15:01:46.867 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:46.867 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880 15:01:46.867 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880.tmp 15:01:46.882 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:46.882 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880 15:01:46.882 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880.tmp 15:01:46.882 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob 19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880 15:01:46.936 DEBUG [standby-run-29] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\19fdc28b49fc2c5ee688a0d9e326bc9750307376d73b1cfa0d63f12c1ca3d44f#5242880.tmp 15:01:47.114 INFO [standby-run-29] StandbyClientSyncExecution.java:82 updated head state successfully: true in 348ms. 15:01:47.114 DEBUG [standby-run-29] StandbyClient.java:157 Channel closed 15:01:49.433 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:01:49.434 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:01:49.434 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:01:49.435 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:01:49.438 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:01:49.501 INFO [main] FileStore.java:471 TarMK closed: target\junit8409575798984762595\junit4975784373500179688 15:01:49.533 INFO [main] FileStore.java:471 TarMK closed: target\junit8409575798984762595\junit7479870917396763324 15:01:49.554 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit3121064064842583521\junit8528459616498256435, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:01:49.554 INFO [main] FileStore.java:240 TarMK opened: target\junit3121064064842583521\junit8528459616498256435 (mmap=false) 15:01:49.570 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit3121064064842583521\junit6868607071162134108, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:01:49.570 INFO [main] FileStore.java:240 TarMK opened: target\junit3121064064842583521\junit6868607071162134108 (mmap=false) 15:01:49.616 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:01:49.616 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:01:49.739 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x6918b68e]' 15:01:49.739 DEBUG [main] StandbyServer.java:249 Binding was successful 15:01:49.917 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client de4e1262-9d0c-4b29-84c4-37e9c3d92714 for current head 15:01:49.917 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:58743 is allowed 15:01:49.917 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:01:49.917 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.917 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.917 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 71fdd401-c4d9-4129-a607-b3abaf0e585f.0000000c to client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.917 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:01:49.917 DEBUG [standby-run-30] StandbyClientSyncExecution.java:103 Found missing segment 71fdd401-c4d9-4129-a607-b3abaf0e585f 15:01:49.917 DEBUG [standby-run-30] StandbyClientSyncExecution.java:124 Inspecting segment 71fdd401-c4d9-4129-a607-b3abaf0e585f 15:01:49.917 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client de4e1262-9d0c-4b29-84c4-37e9c3d92714 for references of segment 71fdd401-c4d9-4129-a607-b3abaf0e585f 15:01:49.917 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:01:49.917 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 71fdd401-c4d9-4129-a607-b3abaf0e585f for client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.917 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 71fdd401-c4d9-4129-a607-b3abaf0e585f to client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.917 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:01:49.932 DEBUG [standby-run-30] StandbyClientSyncExecution.java:184 Found reference from 71fdd401-c4d9-4129-a607-b3abaf0e585f to 92610aa6-fc0a-46aa-a1c7-959683ff796b 15:01:49.932 DEBUG [standby-run-30] StandbyClientSyncExecution.java:124 Inspecting segment 92610aa6-fc0a-46aa-a1c7-959683ff796b 15:01:49.933 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client de4e1262-9d0c-4b29-84c4-37e9c3d92714 for references of segment 92610aa6-fc0a-46aa-a1c7-959683ff796b 15:01:49.934 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:01:49.934 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 92610aa6-fc0a-46aa-a1c7-959683ff796b for client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.935 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 92610aa6-fc0a-46aa-a1c7-959683ff796b to client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.935 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:01:49.936 INFO [standby-run-30] StandbyClientSyncExecution.java:196 Copying data segment 92610aa6-fc0a-46aa-a1c7-959683ff796b from primary 15:01:49.936 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client de4e1262-9d0c-4b29-84c4-37e9c3d92714 for segment 92610aa6-fc0a-46aa-a1c7-959683ff796b 15:01:49.937 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:01:49.937 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.938 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 92610aa6-fc0a-46aa-a1c7-959683ff796b for client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.938 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 192 sent to client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.938 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 92610aa6-fc0a-46aa-a1c7-959683ff796b to client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.939 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:01:49.939 INFO [standby-run-30] StandbyClientSyncExecution.java:196 Copying data segment 71fdd401-c4d9-4129-a607-b3abaf0e585f from primary 15:01:49.939 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client de4e1262-9d0c-4b29-84c4-37e9c3d92714 for segment 71fdd401-c4d9-4129-a607-b3abaf0e585f 15:01:49.939 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:01:49.939 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.939 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 71fdd401-c4d9-4129-a607-b3abaf0e585f for client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.939 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 448 sent to client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.939 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 71fdd401-c4d9-4129-a607-b3abaf0e585f to client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.939 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:01:49.939 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client de4e1262-9d0c-4b29-84c4-37e9c3d92714 for blob dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880 15:01:49.939 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 15:01:49.939 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get blob id' received from client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.939 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880 for client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.939 DEBUG [primary-1] CommunicationObserver.java:105 Binary with size 5242880 sent to client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.939 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880 to client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.954 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880 to client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.986 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880 to client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:49.986 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:49.986 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880 15:01:49.986 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880.tmp 15:01:50.001 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880 to client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:50.001 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:50.001 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880 15:01:50.001 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880.tmp 15:01:50.001 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880 to client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:50.017 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:50.017 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880 15:01:50.017 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880.tmp 15:01:50.017 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880 to client de4e1262-9d0c-4b29-84c4-37e9c3d92714 15:01:50.036 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:50.036 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880 15:01:50.038 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880.tmp 15:01:50.039 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:50.039 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880 15:01:50.039 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880.tmp 15:01:50.055 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880 15:01:50.117 DEBUG [standby-run-30] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\dd17ac1c1e2390682ca21b9e5ede138ebdff0dc15c867e4a4a9d0753efc83731#5242880.tmp 15:01:50.217 INFO [standby-run-30] StandbyClientSyncExecution.java:82 updated head state successfully: true in 300ms. 15:01:50.217 DEBUG [standby-run-30] StandbyClient.java:157 Channel closed 15:01:52.436 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:01:52.437 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:01:52.438 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:01:52.440 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:01:52.440 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:01:52.502 INFO [main] FileStore.java:471 TarMK closed: target\junit3121064064842583521\junit6868607071162134108 15:01:52.540 INFO [main] FileStore.java:471 TarMK closed: target\junit3121064064842583521\junit8528459616498256435 15:01:52.556 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit3425927790952948469\junit4512687301283584923, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:01:52.556 INFO [main] FileStore.java:240 TarMK opened: target\junit3425927790952948469\junit4512687301283584923 (mmap=false) 15:01:52.556 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit3425927790952948469\junit1282713659824980735, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:01:52.556 INFO [main] FileStore.java:240 TarMK opened: target\junit3425927790952948469\junit1282713659824980735 (mmap=false) 15:01:52.603 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:01:52.603 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:01:52.687 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=150000, skip=0,0 15:01:52.718 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x97f246c1]' 15:01:52.718 DEBUG [main] StandbyServer.java:249 Binding was successful 15:01:52.902 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client c46ce126-970b-4109-97a4-ebdd6ec32bf7 for current head 15:01:52.940 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:58830 is allowed 15:01:52.940 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:01:52.940 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.940 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.940 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head fd231c0a-bce0-4d04-a7dc-a1c06bb859d3.0000000c to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.940 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:01:52.940 DEBUG [standby-run-31] StandbyClientSyncExecution.java:103 Found missing segment fd231c0a-bce0-4d04-a7dc-a1c06bb859d3 15:01:52.940 DEBUG [standby-run-31] StandbyClientSyncExecution.java:124 Inspecting segment fd231c0a-bce0-4d04-a7dc-a1c06bb859d3 15:01:52.956 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client c46ce126-970b-4109-97a4-ebdd6ec32bf7 for references of segment fd231c0a-bce0-4d04-a7dc-a1c06bb859d3 15:01:52.956 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:01:52.956 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment fd231c0a-bce0-4d04-a7dc-a1c06bb859d3 for client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.956 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment fd231c0a-bce0-4d04-a7dc-a1c06bb859d3 to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.956 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:01:52.956 DEBUG [standby-run-31] StandbyClientSyncExecution.java:184 Found reference from fd231c0a-bce0-4d04-a7dc-a1c06bb859d3 to f18bf8c1-791e-4917-a423-ed376a9a1176 15:01:52.956 DEBUG [standby-run-31] StandbyClientSyncExecution.java:124 Inspecting segment f18bf8c1-791e-4917-a423-ed376a9a1176 15:01:52.956 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client c46ce126-970b-4109-97a4-ebdd6ec32bf7 for references of segment f18bf8c1-791e-4917-a423-ed376a9a1176 15:01:52.956 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:01:52.956 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment f18bf8c1-791e-4917-a423-ed376a9a1176 for client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.956 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment f18bf8c1-791e-4917-a423-ed376a9a1176 to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.956 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:01:52.956 INFO [standby-run-31] StandbyClientSyncExecution.java:196 Copying data segment f18bf8c1-791e-4917-a423-ed376a9a1176 from primary 15:01:52.956 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client c46ce126-970b-4109-97a4-ebdd6ec32bf7 for segment f18bf8c1-791e-4917-a423-ed376a9a1176 15:01:52.956 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:01:52.956 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.956 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment f18bf8c1-791e-4917-a423-ed376a9a1176 for client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.956 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 192 sent to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.956 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment f18bf8c1-791e-4917-a423-ed376a9a1176 to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.956 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:01:52.956 INFO [standby-run-31] StandbyClientSyncExecution.java:196 Copying data segment fd231c0a-bce0-4d04-a7dc-a1c06bb859d3 from primary 15:01:52.956 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client c46ce126-970b-4109-97a4-ebdd6ec32bf7 for segment fd231c0a-bce0-4d04-a7dc-a1c06bb859d3 15:01:52.956 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:01:52.956 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.956 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment fd231c0a-bce0-4d04-a7dc-a1c06bb859d3 for client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.956 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 448 sent to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.956 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment fd231c0a-bce0-4d04-a7dc-a1c06bb859d3 to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.956 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:01:52.956 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client c46ce126-970b-4109-97a4-ebdd6ec32bf7 for blob 04e5a57aa920f861e2759deffcab1a961ec4dc123304cf76b601984975c3dea2#5242880 15:01:52.956 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 15:01:52.956 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get blob id' received from client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.956 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob 04e5a57aa920f861e2759deffcab1a961ec4dc123304cf76b601984975c3dea2#5242880 for client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.956 DEBUG [primary-1] CommunicationObserver.java:105 Binary with size 5242880 sent to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.956 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob 04e5a57aa920f861e2759deffcab1a961ec4dc123304cf76b601984975c3dea2#5242880 to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.956 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 04e5a57aa920f861e2759deffcab1a961ec4dc123304cf76b601984975c3dea2#5242880 to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.971 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client c46ce126-970b-4109-97a4-ebdd6ec32bf7 io.netty.handler.codec.compression.DecompressionException: mismatching checksum: d9dd1b7e (expected: a28224a0) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] 15:01:52.971 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 04e5a57aa920f861e2759deffcab1a961ec4dc123304cf76b601984975c3dea2#5242880 to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:52.987 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 04e5a57aa920f861e2759deffcab1a961ec4dc123304cf76b601984975c3dea2#5242880 to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:53.003 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 04e5a57aa920f861e2759deffcab1a961ec4dc123304cf76b601984975c3dea2#5242880 to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:53.018 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 04e5a57aa920f861e2759deffcab1a961ec4dc123304cf76b601984975c3dea2#5242880 to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:57.974 DEBUG [standby-run-31] StandbyClient.java:157 Channel closed 15:01:57.974 ERROR [standby-run-31] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to load remote blob 04e5a57aa920f861e2759deffcab1a961ec4dc123304cf76b601984975c3dea2#5242880 at /root/server/#testBlob in 5000ms. Please increase the timeout and try again. at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchAndStoreBlob(StandbyDiff.java:208) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:196) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:181) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:169) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeAdded(StandbyDiff.java:94) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeAdded(CancelableDiff.java:76) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:562) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:566) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:101) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:340) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByteChange2(DataStoreTestBase.java:319) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_60] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_60] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_60] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_60] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121) [surefire-booter-2.19.1.jar:2.19.1] 15:01:57.974 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=-1, skip=-1,0 15:01:58.174 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client c46ce126-970b-4109-97a4-ebdd6ec32bf7 for current head 15:01:58.205 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:58848 is allowed 15:01:58.205 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 15:01:58.205 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get head' received from client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:58.221 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:58.221 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 221ae3ed-6779-4a95-a3ed-0e2ce5834f63.00000006 to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:58.221 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 15:01:58.221 DEBUG [standby-run-32] StandbyClientSyncExecution.java:103 Found missing segment 221ae3ed-6779-4a95-a3ed-0e2ce5834f63 15:01:58.221 DEBUG [standby-run-32] StandbyClientSyncExecution.java:124 Inspecting segment 221ae3ed-6779-4a95-a3ed-0e2ce5834f63 15:01:58.221 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client c46ce126-970b-4109-97a4-ebdd6ec32bf7 for references of segment 221ae3ed-6779-4a95-a3ed-0e2ce5834f63 15:01:58.221 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:01:58.221 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 221ae3ed-6779-4a95-a3ed-0e2ce5834f63 for client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:58.221 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 221ae3ed-6779-4a95-a3ed-0e2ce5834f63 to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:58.221 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:01:58.221 INFO [standby-run-32] StandbyClientSyncExecution.java:196 Copying data segment 221ae3ed-6779-4a95-a3ed-0e2ce5834f63 from primary 15:01:58.221 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client c46ce126-970b-4109-97a4-ebdd6ec32bf7 for segment 221ae3ed-6779-4a95-a3ed-0e2ce5834f63 15:01:58.221 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:01:58.221 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:58.221 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 221ae3ed-6779-4a95-a3ed-0e2ce5834f63 for client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:58.221 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 352 sent to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:58.221 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 221ae3ed-6779-4a95-a3ed-0e2ce5834f63 to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:58.221 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:01:58.221 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client c46ce126-970b-4109-97a4-ebdd6ec32bf7 for blob 31da7a6134601d3356dd0e36b2ec6d9c8f1ad5d1d7feeaff9099ae56ed6c444d#2097152 15:01:58.221 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 15:01:58.221 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get blob id' received from client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:58.221 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 31da7a6134601d3356dd0e36b2ec6d9c8f1ad5d1d7feeaff9099ae56ed6c444d#2097152 for client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:58.221 DEBUG [primary-2] CommunicationObserver.java:105 Binary with size 2097152 sent to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:58.221 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 31da7a6134601d3356dd0e36b2ec6d9c8f1ad5d1d7feeaff9099ae56ed6c444d#2097152 to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:58.221 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/2 of size 1048576 from blob 31da7a6134601d3356dd0e36b2ec6d9c8f1ad5d1d7feeaff9099ae56ed6c444d#2097152 to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:58.243 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/2 of size 1048576 from blob 31da7a6134601d3356dd0e36b2ec6d9c8f1ad5d1d7feeaff9099ae56ed6c444d#2097152 to client c46ce126-970b-4109-97a4-ebdd6ec32bf7 15:01:58.243 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:58.259 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/2 of size 1048576 from blob 31da7a6134601d3356dd0e36b2ec6d9c8f1ad5d1d7feeaff9099ae56ed6c444d#2097152 15:01:58.259 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\31da7a6134601d3356dd0e36b2ec6d9c8f1ad5d1d7feeaff9099ae56ed6c444d#2097152.tmp 15:01:58.274 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:01:58.274 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/2 of size 1048576 from blob 31da7a6134601d3356dd0e36b2ec6d9c8f1ad5d1d7feeaff9099ae56ed6c444d#2097152 15:01:58.274 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\31da7a6134601d3356dd0e36b2ec6d9c8f1ad5d1d7feeaff9099ae56ed6c444d#2097152.tmp 15:01:58.274 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob 31da7a6134601d3356dd0e36b2ec6d9c8f1ad5d1d7feeaff9099ae56ed6c444d#2097152 15:01:58.306 DEBUG [standby-run-32] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\31da7a6134601d3356dd0e36b2ec6d9c8f1ad5d1d7feeaff9099ae56ed6c444d#2097152.tmp 15:01:58.421 INFO [standby-run-32] StandbyClientSyncExecution.java:82 updated head state successfully: true in 200ms. 15:01:58.421 DEBUG [standby-run-32] StandbyClient.java:157 Channel closed 15:02:00.802 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:02:00.802 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:02:00.802 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:02:00.802 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:02:00.802 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:02:00.886 INFO [main] FileStore.java:471 TarMK closed: target\junit3425927790952948469\junit1282713659824980735 15:02:00.902 INFO [main] FileStore.java:471 TarMK closed: target\junit3425927790952948469\junit4512687301283584923 15:02:00.933 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit8827897774949739135\junit4273175169211585243, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:02:00.933 INFO [main] FileStore.java:240 TarMK opened: target\junit8827897774949739135\junit4273175169211585243 (mmap=false) 15:02:00.933 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit8827897774949739135\junit7241179345819981300, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:02:00.933 INFO [main] FileStore.java:240 TarMK opened: target\junit8827897774949739135\junit7241179345819981300 (mmap=false) 15:02:00.986 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:02:00.986 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:02:01.071 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=150, skip=0,0 15:02:01.102 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xd3a5419d]' 15:02:01.102 DEBUG [main] StandbyServer.java:249 Binding was successful 15:02:01.286 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 5e38b8c5-3470-40e8-859f-06ec9e27a418 for current head 15:02:01.333 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:58934 is allowed 15:02:01.333 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:02:01.333 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:01.333 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:01.333 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 5299f42d-2238-438b-ab78-4fdea4f131c8.0000000c to client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:01.333 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:02:01.333 DEBUG [standby-run-33] StandbyClientSyncExecution.java:103 Found missing segment 5299f42d-2238-438b-ab78-4fdea4f131c8 15:02:01.333 DEBUG [standby-run-33] StandbyClientSyncExecution.java:124 Inspecting segment 5299f42d-2238-438b-ab78-4fdea4f131c8 15:02:01.333 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 5e38b8c5-3470-40e8-859f-06ec9e27a418 for references of segment 5299f42d-2238-438b-ab78-4fdea4f131c8 15:02:01.333 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:02:01.333 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 5299f42d-2238-438b-ab78-4fdea4f131c8 for client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:01.333 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 5299f42d-2238-438b-ab78-4fdea4f131c8 to client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:01.349 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 5e38b8c5-3470-40e8-859f-06ec9e27a418 io.netty.handler.codec.compression.DecompressionException: mismatching checksum: a282481a (expected: a2829238) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] 15:02:06.354 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 5e38b8c5-3470-40e8-859f-06ec9e27a418 io.netty.handler.timeout.ReadTimeoutException: null 15:02:06.354 DEBUG [standby-run-33] StandbyClient.java:157 Channel closed 15:02:06.354 ERROR [standby-run-33] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment 5299f42d-2238-438b-ab78-4fdea4f131c8 from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:340) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByteChange(DataStoreTestBase.java:314) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_60] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_60] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_60] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_60] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121) [surefire-booter-2.19.1.jar:2.19.1] 15:02:06.354 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=-1, skip=-1,0 15:02:06.554 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client 5e38b8c5-3470-40e8-859f-06ec9e27a418 for current head 15:02:06.592 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:58952 is allowed 15:02:06.592 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 15:02:06.592 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get head' received from client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.592 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.592 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 3b36541d-31ff-4c10-a5d6-f5e850749780.00000006 to client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.592 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 15:02:06.592 DEBUG [standby-run-34] StandbyClientSyncExecution.java:103 Found missing segment 3b36541d-31ff-4c10-a5d6-f5e850749780 15:02:06.592 DEBUG [standby-run-34] StandbyClientSyncExecution.java:124 Inspecting segment 3b36541d-31ff-4c10-a5d6-f5e850749780 15:02:06.592 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 5e38b8c5-3470-40e8-859f-06ec9e27a418 for references of segment 3b36541d-31ff-4c10-a5d6-f5e850749780 15:02:06.592 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:02:06.592 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 3b36541d-31ff-4c10-a5d6-f5e850749780 for client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.592 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 3b36541d-31ff-4c10-a5d6-f5e850749780 to client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.592 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:02:06.592 DEBUG [standby-run-34] StandbyClientSyncExecution.java:184 Found reference from 3b36541d-31ff-4c10-a5d6-f5e850749780 to 5299f42d-2238-438b-ab78-4fdea4f131c8 15:02:06.607 DEBUG [standby-run-34] StandbyClientSyncExecution.java:184 Found reference from 3b36541d-31ff-4c10-a5d6-f5e850749780 to c4c11634-6c38-4876-a08d-3523b966821b 15:02:06.607 DEBUG [standby-run-34] StandbyClientSyncExecution.java:124 Inspecting segment 5299f42d-2238-438b-ab78-4fdea4f131c8 15:02:06.607 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 5e38b8c5-3470-40e8-859f-06ec9e27a418 for references of segment 5299f42d-2238-438b-ab78-4fdea4f131c8 15:02:06.607 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:02:06.607 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 5299f42d-2238-438b-ab78-4fdea4f131c8 for client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.607 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 5299f42d-2238-438b-ab78-4fdea4f131c8 to client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.607 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:02:06.607 DEBUG [standby-run-34] StandbyClientSyncExecution.java:124 Inspecting segment c4c11634-6c38-4876-a08d-3523b966821b 15:02:06.607 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 5e38b8c5-3470-40e8-859f-06ec9e27a418 for references of segment c4c11634-6c38-4876-a08d-3523b966821b 15:02:06.607 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:02:06.607 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment c4c11634-6c38-4876-a08d-3523b966821b for client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.607 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment c4c11634-6c38-4876-a08d-3523b966821b to client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.607 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:02:06.607 INFO [standby-run-34] StandbyClientSyncExecution.java:196 Copying data segment c4c11634-6c38-4876-a08d-3523b966821b from primary 15:02:06.607 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 5e38b8c5-3470-40e8-859f-06ec9e27a418 for segment c4c11634-6c38-4876-a08d-3523b966821b 15:02:06.607 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:02:06.607 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.607 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment c4c11634-6c38-4876-a08d-3523b966821b for client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.607 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 192 sent to client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.607 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment c4c11634-6c38-4876-a08d-3523b966821b to client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.607 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:02:06.607 INFO [standby-run-34] StandbyClientSyncExecution.java:196 Copying data segment 5299f42d-2238-438b-ab78-4fdea4f131c8 from primary 15:02:06.607 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 5e38b8c5-3470-40e8-859f-06ec9e27a418 for segment 5299f42d-2238-438b-ab78-4fdea4f131c8 15:02:06.607 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:02:06.607 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.607 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 5299f42d-2238-438b-ab78-4fdea4f131c8 for client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.607 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 448 sent to client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.607 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 5299f42d-2238-438b-ab78-4fdea4f131c8 to client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.607 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:02:06.607 INFO [standby-run-34] StandbyClientSyncExecution.java:196 Copying data segment 3b36541d-31ff-4c10-a5d6-f5e850749780 from primary 15:02:06.607 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 5e38b8c5-3470-40e8-859f-06ec9e27a418 for segment 3b36541d-31ff-4c10-a5d6-f5e850749780 15:02:06.607 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:02:06.607 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.607 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 3b36541d-31ff-4c10-a5d6-f5e850749780 for client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.607 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 352 sent to client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.607 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 3b36541d-31ff-4c10-a5d6-f5e850749780 to client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.607 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:02:06.607 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client 5e38b8c5-3470-40e8-859f-06ec9e27a418 for blob 4be4ed2687da77a875ec5aee2350ada843784b0c98b77c733285533b629ec5b4#2097152 15:02:06.607 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 15:02:06.607 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get blob id' received from client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.607 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 4be4ed2687da77a875ec5aee2350ada843784b0c98b77c733285533b629ec5b4#2097152 for client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.607 DEBUG [primary-2] CommunicationObserver.java:105 Binary with size 2097152 sent to client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.607 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 4be4ed2687da77a875ec5aee2350ada843784b0c98b77c733285533b629ec5b4#2097152 to client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.623 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/2 of size 1048576 from blob 4be4ed2687da77a875ec5aee2350ada843784b0c98b77c733285533b629ec5b4#2097152 to client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.639 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/2 of size 1048576 from blob 4be4ed2687da77a875ec5aee2350ada843784b0c98b77c733285533b629ec5b4#2097152 to client 5e38b8c5-3470-40e8-859f-06ec9e27a418 15:02:06.639 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:02:06.639 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/2 of size 1048576 from blob 4be4ed2687da77a875ec5aee2350ada843784b0c98b77c733285533b629ec5b4#2097152 15:02:06.639 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4be4ed2687da77a875ec5aee2350ada843784b0c98b77c733285533b629ec5b4#2097152.tmp 15:02:06.654 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:02:06.654 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/2 of size 1048576 from blob 4be4ed2687da77a875ec5aee2350ada843784b0c98b77c733285533b629ec5b4#2097152 15:02:06.654 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4be4ed2687da77a875ec5aee2350ada843784b0c98b77c733285533b629ec5b4#2097152.tmp 15:02:06.654 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob 4be4ed2687da77a875ec5aee2350ada843784b0c98b77c733285533b629ec5b4#2097152 15:02:06.676 DEBUG [standby-run-34] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\4be4ed2687da77a875ec5aee2350ada843784b0c98b77c733285533b629ec5b4#2097152.tmp 15:02:06.807 INFO [standby-run-34] StandbyClientSyncExecution.java:82 updated head state successfully: true in 215ms. 15:02:06.807 DEBUG [standby-run-34] StandbyClient.java:157 Channel closed 15:02:09.126 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:02:09.126 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:02:09.126 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:02:09.126 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:02:09.126 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:02:09.157 INFO [main] FileStore.java:471 TarMK closed: target\junit8827897774949739135\junit7241179345819981300 15:02:09.179 INFO [main] FileStore.java:471 TarMK closed: target\junit8827897774949739135\junit4273175169211585243 15:02:09.210 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1818328095020329597\junit6329451746702053091, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:02:09.210 INFO [main] FileStore.java:240 TarMK opened: target\junit1818328095020329597\junit6329451746702053091 (mmap=false) 15:02:09.210 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1818328095020329597\junit4286585837449045692, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:02:09.226 INFO [main] FileStore.java:240 TarMK opened: target\junit1818328095020329597\junit4286585837449045692 (mmap=false) 15:02:09.279 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:02:09.279 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:02:09.379 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=-1, skip=100,1 15:02:09.411 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xdf8871bf]' 15:02:09.411 DEBUG [main] StandbyServer.java:249 Binding was successful 15:02:09.577 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client a42688b1-ab1e-410d-893d-1f21a72f163c for current head 15:02:09.611 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:59038 is allowed 15:02:09.611 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:02:09.611 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:09.611 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:09.611 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 851c414d-3a0c-4194-a29e-ef6b4feeb53c.0000000c to client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:09.626 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:02:09.626 DEBUG [standby-run-35] StandbyClientSyncExecution.java:103 Found missing segment 851c414d-3a0c-4194-a29e-ef6b4feeb53c 15:02:09.626 DEBUG [standby-run-35] StandbyClientSyncExecution.java:124 Inspecting segment 851c414d-3a0c-4194-a29e-ef6b4feeb53c 15:02:09.626 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client a42688b1-ab1e-410d-893d-1f21a72f163c for references of segment 851c414d-3a0c-4194-a29e-ef6b4feeb53c 15:02:09.626 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:02:09.626 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 851c414d-3a0c-4194-a29e-ef6b4feeb53c for client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:09.626 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 851c414d-3a0c-4194-a29e-ef6b4feeb53c to client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.629 DEBUG [standby-run-35] StandbyClient.java:157 Channel closed 15:02:14.629 ERROR [standby-run-35] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment 851c414d-3a0c-4194-a29e-ef6b4feeb53c from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:340) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxySkippedBytesIntermediateChange(DataStoreTestBase.java:294) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_60] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_60] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_60] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_60] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121) [surefire-booter-2.19.1.jar:2.19.1] 15:02:14.629 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=-1, skip=-1,0 15:02:14.798 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client a42688b1-ab1e-410d-893d-1f21a72f163c for current head 15:02:14.844 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:59056 is allowed 15:02:14.844 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 15:02:14.844 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get head' received from client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.844 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.844 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head e7cbfea8-8737-46ef-a95d-bdf33dc58deb.00000006 to client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.844 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 15:02:14.844 DEBUG [standby-run-36] StandbyClientSyncExecution.java:103 Found missing segment e7cbfea8-8737-46ef-a95d-bdf33dc58deb 15:02:14.844 DEBUG [standby-run-36] StandbyClientSyncExecution.java:124 Inspecting segment e7cbfea8-8737-46ef-a95d-bdf33dc58deb 15:02:14.844 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client a42688b1-ab1e-410d-893d-1f21a72f163c for references of segment e7cbfea8-8737-46ef-a95d-bdf33dc58deb 15:02:14.844 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:02:14.844 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment e7cbfea8-8737-46ef-a95d-bdf33dc58deb for client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.844 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment e7cbfea8-8737-46ef-a95d-bdf33dc58deb to client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.844 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:02:14.844 DEBUG [standby-run-36] StandbyClientSyncExecution.java:184 Found reference from e7cbfea8-8737-46ef-a95d-bdf33dc58deb to 851c414d-3a0c-4194-a29e-ef6b4feeb53c 15:02:14.844 DEBUG [standby-run-36] StandbyClientSyncExecution.java:184 Found reference from e7cbfea8-8737-46ef-a95d-bdf33dc58deb to f98cc765-f4d3-4150-a425-9172d5401e93 15:02:14.844 DEBUG [standby-run-36] StandbyClientSyncExecution.java:124 Inspecting segment 851c414d-3a0c-4194-a29e-ef6b4feeb53c 15:02:14.844 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client a42688b1-ab1e-410d-893d-1f21a72f163c for references of segment 851c414d-3a0c-4194-a29e-ef6b4feeb53c 15:02:14.844 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:02:14.844 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 851c414d-3a0c-4194-a29e-ef6b4feeb53c for client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.844 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 851c414d-3a0c-4194-a29e-ef6b4feeb53c to client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.844 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:02:14.844 DEBUG [standby-run-36] StandbyClientSyncExecution.java:124 Inspecting segment f98cc765-f4d3-4150-a425-9172d5401e93 15:02:14.844 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client a42688b1-ab1e-410d-893d-1f21a72f163c for references of segment f98cc765-f4d3-4150-a425-9172d5401e93 15:02:14.844 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:02:14.844 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment f98cc765-f4d3-4150-a425-9172d5401e93 for client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.844 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment f98cc765-f4d3-4150-a425-9172d5401e93 to client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.844 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:02:14.844 INFO [standby-run-36] StandbyClientSyncExecution.java:196 Copying data segment f98cc765-f4d3-4150-a425-9172d5401e93 from primary 15:02:14.844 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client a42688b1-ab1e-410d-893d-1f21a72f163c for segment f98cc765-f4d3-4150-a425-9172d5401e93 15:02:14.844 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:02:14.844 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.844 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment f98cc765-f4d3-4150-a425-9172d5401e93 for client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.844 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 192 sent to client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.844 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment f98cc765-f4d3-4150-a425-9172d5401e93 to client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.860 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:02:14.860 INFO [standby-run-36] StandbyClientSyncExecution.java:196 Copying data segment 851c414d-3a0c-4194-a29e-ef6b4feeb53c from primary 15:02:14.860 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client a42688b1-ab1e-410d-893d-1f21a72f163c for segment 851c414d-3a0c-4194-a29e-ef6b4feeb53c 15:02:14.860 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:02:14.860 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.860 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 851c414d-3a0c-4194-a29e-ef6b4feeb53c for client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.860 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 448 sent to client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.860 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 851c414d-3a0c-4194-a29e-ef6b4feeb53c to client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.860 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:02:14.860 INFO [standby-run-36] StandbyClientSyncExecution.java:196 Copying data segment e7cbfea8-8737-46ef-a95d-bdf33dc58deb from primary 15:02:14.860 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client a42688b1-ab1e-410d-893d-1f21a72f163c for segment e7cbfea8-8737-46ef-a95d-bdf33dc58deb 15:02:14.860 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:02:14.860 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.860 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment e7cbfea8-8737-46ef-a95d-bdf33dc58deb for client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.860 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 352 sent to client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.860 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment e7cbfea8-8737-46ef-a95d-bdf33dc58deb to client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.860 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:02:14.860 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client a42688b1-ab1e-410d-893d-1f21a72f163c for blob 56940776118958e9edae686d9d3cfb59de6f0fec74b8cde0c1b4611c1f003b27#2097152 15:02:14.860 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 15:02:14.860 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get blob id' received from client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.860 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 56940776118958e9edae686d9d3cfb59de6f0fec74b8cde0c1b4611c1f003b27#2097152 for client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.860 DEBUG [primary-2] CommunicationObserver.java:105 Binary with size 2097152 sent to client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.860 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 56940776118958e9edae686d9d3cfb59de6f0fec74b8cde0c1b4611c1f003b27#2097152 to client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.860 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/2 of size 1048576 from blob 56940776118958e9edae686d9d3cfb59de6f0fec74b8cde0c1b4611c1f003b27#2097152 to client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.882 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/2 of size 1048576 from blob 56940776118958e9edae686d9d3cfb59de6f0fec74b8cde0c1b4611c1f003b27#2097152 to client a42688b1-ab1e-410d-893d-1f21a72f163c 15:02:14.882 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:02:14.882 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/2 of size 1048576 from blob 56940776118958e9edae686d9d3cfb59de6f0fec74b8cde0c1b4611c1f003b27#2097152 15:02:14.882 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\56940776118958e9edae686d9d3cfb59de6f0fec74b8cde0c1b4611c1f003b27#2097152.tmp 15:02:14.898 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:02:14.898 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/2 of size 1048576 from blob 56940776118958e9edae686d9d3cfb59de6f0fec74b8cde0c1b4611c1f003b27#2097152 15:02:14.898 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\56940776118958e9edae686d9d3cfb59de6f0fec74b8cde0c1b4611c1f003b27#2097152.tmp 15:02:14.898 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob 56940776118958e9edae686d9d3cfb59de6f0fec74b8cde0c1b4611c1f003b27#2097152 15:02:14.929 DEBUG [standby-run-36] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\56940776118958e9edae686d9d3cfb59de6f0fec74b8cde0c1b4611c1f003b27#2097152.tmp 15:02:15.060 INFO [standby-run-36] StandbyClientSyncExecution.java:82 updated head state successfully: true in 216ms. 15:02:15.060 DEBUG [standby-run-36] StandbyClient.java:157 Channel closed 15:02:17.378 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:02:17.379 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:02:17.380 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:02:17.381 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:02:17.384 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:02:17.478 INFO [main] FileStore.java:471 TarMK closed: target\junit1818328095020329597\junit4286585837449045692 15:02:17.500 INFO [main] FileStore.java:471 TarMK closed: target\junit1818328095020329597\junit6329451746702053091 15:02:17.515 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit8076342022685316856\junit1851125640046590150, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:02:17.515 INFO [main] FileStore.java:240 TarMK opened: target\junit8076342022685316856\junit1851125640046590150 (mmap=false) 15:02:17.531 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit8076342022685316856\junit4374479655624931309, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:02:17.531 INFO [main] FileStore.java:240 TarMK opened: target\junit8076342022685316856\junit4374479655624931309 (mmap=false) 15:02:17.581 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:02:17.582 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:02:17.663 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=150, skip=0,0 15:02:17.685 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x4c7ce824]' 15:02:17.700 DEBUG [main] StandbyServer.java:249 Binding was successful 15:02:17.832 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client c112521f-76f5-49fe-b098-0a1cc645f7ca for current head 15:02:17.885 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:59142 is allowed 15:02:17.885 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:02:17.885 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:17.885 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:17.885 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 49aab673-1c36-4691-a9f3-620f8acca2de.0000000c to client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:17.885 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:02:17.885 DEBUG [standby-run-37] StandbyClientSyncExecution.java:103 Found missing segment 49aab673-1c36-4691-a9f3-620f8acca2de 15:02:17.885 DEBUG [standby-run-37] StandbyClientSyncExecution.java:124 Inspecting segment 49aab673-1c36-4691-a9f3-620f8acca2de 15:02:17.885 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client c112521f-76f5-49fe-b098-0a1cc645f7ca for references of segment 49aab673-1c36-4691-a9f3-620f8acca2de 15:02:17.885 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:02:17.885 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 49aab673-1c36-4691-a9f3-620f8acca2de for client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:17.885 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 49aab673-1c36-4691-a9f3-620f8acca2de to client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:17.885 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client c112521f-76f5-49fe-b098-0a1cc645f7ca io.netty.handler.codec.compression.DecompressionException: mismatching checksum: 11044b (expected: a2822110) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] 15:02:22.904 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client c112521f-76f5-49fe-b098-0a1cc645f7ca io.netty.handler.timeout.ReadTimeoutException: null 15:02:22.904 DEBUG [standby-run-37] StandbyClient.java:157 Channel closed 15:02:22.904 ERROR [standby-run-37] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment 49aab673-1c36-4691-a9f3-620f8acca2de from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:340) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByte(DataStoreTestBase.java:304) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_60] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_60] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_60] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_60] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121) [surefire-booter-2.19.1.jar:2.19.1] 15:02:22.904 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=-1, skip=-1,0 15:02:22.920 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client c112521f-76f5-49fe-b098-0a1cc645f7ca for current head 15:02:22.951 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:59160 is allowed 15:02:22.951 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 15:02:22.951 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get head' received from client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.951 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.951 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 49aab673-1c36-4691-a9f3-620f8acca2de.0000000c to client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.966 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 15:02:22.966 DEBUG [standby-run-38] StandbyClientSyncExecution.java:103 Found missing segment 49aab673-1c36-4691-a9f3-620f8acca2de 15:02:22.966 DEBUG [standby-run-38] StandbyClientSyncExecution.java:124 Inspecting segment 49aab673-1c36-4691-a9f3-620f8acca2de 15:02:22.966 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client c112521f-76f5-49fe-b098-0a1cc645f7ca for references of segment 49aab673-1c36-4691-a9f3-620f8acca2de 15:02:22.966 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:02:22.966 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 49aab673-1c36-4691-a9f3-620f8acca2de for client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.966 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 49aab673-1c36-4691-a9f3-620f8acca2de to client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.966 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:02:22.966 DEBUG [standby-run-38] StandbyClientSyncExecution.java:184 Found reference from 49aab673-1c36-4691-a9f3-620f8acca2de to d66780e1-d163-46f3-a801-21de4278bc8c 15:02:22.966 DEBUG [standby-run-38] StandbyClientSyncExecution.java:124 Inspecting segment d66780e1-d163-46f3-a801-21de4278bc8c 15:02:22.966 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client c112521f-76f5-49fe-b098-0a1cc645f7ca for references of segment d66780e1-d163-46f3-a801-21de4278bc8c 15:02:22.966 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:02:22.966 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment d66780e1-d163-46f3-a801-21de4278bc8c for client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.966 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment d66780e1-d163-46f3-a801-21de4278bc8c to client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.966 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:02:22.966 INFO [standby-run-38] StandbyClientSyncExecution.java:196 Copying data segment d66780e1-d163-46f3-a801-21de4278bc8c from primary 15:02:22.966 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client c112521f-76f5-49fe-b098-0a1cc645f7ca for segment d66780e1-d163-46f3-a801-21de4278bc8c 15:02:22.966 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:02:22.966 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.966 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment d66780e1-d163-46f3-a801-21de4278bc8c for client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.966 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 192 sent to client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.966 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment d66780e1-d163-46f3-a801-21de4278bc8c to client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.966 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:02:22.966 INFO [standby-run-38] StandbyClientSyncExecution.java:196 Copying data segment 49aab673-1c36-4691-a9f3-620f8acca2de from primary 15:02:22.966 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client c112521f-76f5-49fe-b098-0a1cc645f7ca for segment 49aab673-1c36-4691-a9f3-620f8acca2de 15:02:22.966 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:02:22.966 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.966 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 49aab673-1c36-4691-a9f3-620f8acca2de for client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.966 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 448 sent to client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.966 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 49aab673-1c36-4691-a9f3-620f8acca2de to client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.966 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:02:22.966 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client c112521f-76f5-49fe-b098-0a1cc645f7ca for blob 4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880 15:02:22.966 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 15:02:22.966 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get blob id' received from client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.966 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880 for client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.966 DEBUG [primary-2] CommunicationObserver.java:105 Binary with size 5242880 sent to client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.966 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880 to client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.966 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880 to client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.989 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880 to client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:22.989 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:02:22.989 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880 15:02:22.989 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880.tmp 15:02:22.989 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880 to client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:23.004 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:02:23.004 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880 15:02:23.004 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880.tmp 15:02:23.004 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880 to client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:23.020 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:02:23.020 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880 15:02:23.020 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880.tmp 15:02:23.020 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880 to client c112521f-76f5-49fe-b098-0a1cc645f7ca 15:02:23.035 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:02:23.035 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880 15:02:23.035 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880.tmp 15:02:23.051 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:02:23.051 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880 15:02:23.067 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880.tmp 15:02:23.067 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob 4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880 15:02:23.151 DEBUG [standby-run-38] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\4ca199536226ff95a7ec6a7a217c3dfe9c6c2ee90f9754ea0ae38d6f3b9df78e#5242880.tmp 15:02:23.321 INFO [standby-run-38] StandbyClientSyncExecution.java:82 updated head state successfully: true in 355ms. 15:02:23.321 DEBUG [standby-run-38] StandbyClient.java:157 Channel closed 15:02:25.652 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:02:25.652 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:02:25.652 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:02:25.652 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:02:25.652 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:02:25.705 INFO [main] FileStore.java:471 TarMK closed: target\junit8076342022685316856\junit4374479655624931309 15:02:25.737 INFO [main] FileStore.java:471 TarMK closed: target\junit8076342022685316856\junit1851125640046590150 15:02:25.752 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit3384613363835420517\junit7887467596799006801, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:02:25.752 INFO [main] FileStore.java:240 TarMK opened: target\junit3384613363835420517\junit7887467596799006801 (mmap=false) 15:02:25.789 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit3384613363835420517\junit8371879261624977341, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:02:25.805 INFO [main] FileStore.java:240 TarMK opened: target\junit3384613363835420517\junit8371879261624977341 (mmap=false) 15:02:25.852 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:02:25.852 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:02:25.953 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=150000, skip=0,0 15:02:25.968 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x14f8d9de]' 15:02:25.985 DEBUG [main] StandbyServer.java:249 Binding was successful 15:02:26.438 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 9fa18d55-68ed-440b-9884-469d817d45b4 for current head 15:02:26.486 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:59246 is allowed 15:02:26.486 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:02:26.487 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.487 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.487 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head ec7d5a19-f0e6-4075-a96e-97a4b9bb1b02.0000000c to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.488 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:02:26.489 DEBUG [standby-run-39] StandbyClientSyncExecution.java:103 Found missing segment ec7d5a19-f0e6-4075-a96e-97a4b9bb1b02 15:02:26.489 DEBUG [standby-run-39] StandbyClientSyncExecution.java:124 Inspecting segment ec7d5a19-f0e6-4075-a96e-97a4b9bb1b02 15:02:26.489 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 9fa18d55-68ed-440b-9884-469d817d45b4 for references of segment ec7d5a19-f0e6-4075-a96e-97a4b9bb1b02 15:02:26.490 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:02:26.490 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment ec7d5a19-f0e6-4075-a96e-97a4b9bb1b02 for client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.491 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment ec7d5a19-f0e6-4075-a96e-97a4b9bb1b02 to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.491 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:02:26.491 DEBUG [standby-run-39] StandbyClientSyncExecution.java:184 Found reference from ec7d5a19-f0e6-4075-a96e-97a4b9bb1b02 to 4db76d45-df53-411a-a847-edda58b298b0 15:02:26.491 DEBUG [standby-run-39] StandbyClientSyncExecution.java:124 Inspecting segment 4db76d45-df53-411a-a847-edda58b298b0 15:02:26.491 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 9fa18d55-68ed-440b-9884-469d817d45b4 for references of segment 4db76d45-df53-411a-a847-edda58b298b0 15:02:26.491 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:02:26.491 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 4db76d45-df53-411a-a847-edda58b298b0 for client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.491 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 4db76d45-df53-411a-a847-edda58b298b0 to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.491 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:02:26.491 INFO [standby-run-39] StandbyClientSyncExecution.java:196 Copying data segment 4db76d45-df53-411a-a847-edda58b298b0 from primary 15:02:26.491 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 9fa18d55-68ed-440b-9884-469d817d45b4 for segment 4db76d45-df53-411a-a847-edda58b298b0 15:02:26.491 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:02:26.491 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.491 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 4db76d45-df53-411a-a847-edda58b298b0 for client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.491 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 192 sent to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.491 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 4db76d45-df53-411a-a847-edda58b298b0 to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.491 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:02:26.491 INFO [standby-run-39] StandbyClientSyncExecution.java:196 Copying data segment ec7d5a19-f0e6-4075-a96e-97a4b9bb1b02 from primary 15:02:26.491 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 9fa18d55-68ed-440b-9884-469d817d45b4 for segment ec7d5a19-f0e6-4075-a96e-97a4b9bb1b02 15:02:26.491 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:02:26.491 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.491 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment ec7d5a19-f0e6-4075-a96e-97a4b9bb1b02 for client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.491 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 448 sent to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.491 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment ec7d5a19-f0e6-4075-a96e-97a4b9bb1b02 to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.491 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:02:26.491 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 9fa18d55-68ed-440b-9884-469d817d45b4 for blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 15:02:26.491 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 15:02:26.491 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get blob id' received from client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.491 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 for client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.491 DEBUG [primary-1] CommunicationObserver.java:105 Binary with size 5242880 sent to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.491 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.506 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.506 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 9fa18d55-68ed-440b-9884-469d817d45b4 io.netty.handler.codec.compression.DecompressionException: mismatching checksum: a2822597 (expected: 20971ea9) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] 15:02:26.522 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.522 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.538 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:26.553 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:31.508 DEBUG [standby-run-39] StandbyClient.java:157 Channel closed 15:02:31.508 ERROR [standby-run-39] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to load remote blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 at /root/server/#testBlob in 5000ms. Please increase the timeout and try again. at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchAndStoreBlob(StandbyDiff.java:208) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:196) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:181) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:169) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeAdded(StandbyDiff.java:94) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeAdded(CancelableDiff.java:76) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:562) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:566) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:101) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:340) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByte2(DataStoreTestBase.java:309) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_60] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_60] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_60] at java.lang.reflect.Method.invoke(Method.java:497) ~[na:1.8.0_60] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161) [surefire-junit4-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:290) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:242) [surefire-booter-2.19.1.jar:2.19.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:121) [surefire-booter-2.19.1.jar:2.19.1] 15:02:31.508 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=-1, skip=-1,0 15:02:31.523 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client 9fa18d55-68ed-440b-9884-469d817d45b4 for current head 15:02:31.554 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:59264 is allowed 15:02:31.554 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 15:02:31.554 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get head' received from client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:31.554 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:31.554 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head ec7d5a19-f0e6-4075-a96e-97a4b9bb1b02.0000000c to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:31.554 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 15:02:31.554 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client 9fa18d55-68ed-440b-9884-469d817d45b4 for blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 15:02:31.554 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 15:02:31.554 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get blob id' received from client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:31.554 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 for client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:31.554 DEBUG [primary-2] CommunicationObserver.java:105 Binary with size 5242880 sent to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:31.554 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:31.570 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:31.570 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:31.586 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:02:31.587 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 15:02:31.590 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880.tmp 15:02:31.592 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:31.592 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:02:31.592 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 15:02:31.608 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880.tmp 15:02:31.608 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:31.623 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:02:31.623 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 15:02:31.623 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880.tmp 15:02:31.623 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 to client 9fa18d55-68ed-440b-9884-469d817d45b4 15:02:31.639 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:02:31.639 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 15:02:31.639 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880.tmp 15:02:31.655 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:02:31.655 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 15:02:31.655 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880.tmp 15:02:31.655 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880 15:02:31.708 DEBUG [standby-run-40] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\e86da00223257856358a4f567f97f91e2a26634187fb5905ce90717f883686c4#5242880.tmp 15:02:31.870 INFO [standby-run-40] StandbyClientSyncExecution.java:82 updated head state successfully: true in 316ms. 15:02:31.870 DEBUG [standby-run-40] StandbyClient.java:157 Channel closed 15:02:34.208 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:02:34.208 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:02:34.208 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:02:34.208 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:02:34.208 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:02:34.271 INFO [main] FileStore.java:471 TarMK closed: target\junit3384613363835420517\junit8371879261624977341 15:02:34.340 INFO [main] FileStore.java:471 TarMK closed: target\junit3384613363835420517\junit7887467596799006801 15:02:34.355 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit8277413677521668728\junit6329151603887954162, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:02:34.355 INFO [main] FileStore.java:240 TarMK opened: target\junit8277413677521668728\junit6329151603887954162 (mmap=false) 15:02:34.409 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit8277413677521668728\junit6747363152698468464, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:02:34.409 INFO [main] FileStore.java:240 TarMK opened: target\junit8277413677521668728\junit6747363152698468464 (mmap=false) 15:02:34.471 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:02:34.471 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:02:34.614 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=0, skip=0,0 15:02:34.629 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xe3fc3638]' 15:02:34.629 DEBUG [main] StandbyServer.java:249 Binding was successful 15:02:34.976 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 420f258f-b6b4-4767-858d-0f47bf468a42 for current head 15:02:35.014 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:59351 is allowed 15:02:35.014 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:02:35.014 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:35.014 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:35.014 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 9cd0a741-268d-4bc0-a351-4e315ff45bd4.0000000c to client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:35.014 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 420f258f-b6b4-4767-858d-0f47bf468a42 io.netty.handler.codec.compression.DecompressionException: Received COMPRESSED_DATA tag before STREAM_IDENTIFIER at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:162) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:745) [na:1.8.0_60] 15:02:39.980 ERROR [standby-run-41] StandbyClientSyncExecution.java:67 Unable to fetch remote head 15:02:39.980 DEBUG [standby-run-41] StandbyClient.java:157 Channel closed 15:02:39.980 INFO [main] NetworkErrorProxy.java:80 Starting proxy with flip=-1, skip=-1,0 15:02:39.995 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client 420f258f-b6b4-4767-858d-0f47bf468a42 for current head 15:02:40.018 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:59369 is allowed 15:02:40.033 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 15:02:40.033 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get head' received from client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.033 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.033 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 9cd0a741-268d-4bc0-a351-4e315ff45bd4.0000000c to client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.033 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 15:02:40.033 DEBUG [standby-run-42] StandbyClientSyncExecution.java:103 Found missing segment 9cd0a741-268d-4bc0-a351-4e315ff45bd4 15:02:40.033 DEBUG [standby-run-42] StandbyClientSyncExecution.java:124 Inspecting segment 9cd0a741-268d-4bc0-a351-4e315ff45bd4 15:02:40.033 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 420f258f-b6b4-4767-858d-0f47bf468a42 for references of segment 9cd0a741-268d-4bc0-a351-4e315ff45bd4 15:02:40.033 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:02:40.033 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 9cd0a741-268d-4bc0-a351-4e315ff45bd4 for client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.033 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 9cd0a741-268d-4bc0-a351-4e315ff45bd4 to client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.033 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:02:40.033 DEBUG [standby-run-42] StandbyClientSyncExecution.java:184 Found reference from 9cd0a741-268d-4bc0-a351-4e315ff45bd4 to 1c8bbd41-5c52-4bd7-a823-2fe7b0e2280e 15:02:40.033 DEBUG [standby-run-42] StandbyClientSyncExecution.java:124 Inspecting segment 1c8bbd41-5c52-4bd7-a823-2fe7b0e2280e 15:02:40.033 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 420f258f-b6b4-4767-858d-0f47bf468a42 for references of segment 1c8bbd41-5c52-4bd7-a823-2fe7b0e2280e 15:02:40.033 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 15:02:40.033 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 1c8bbd41-5c52-4bd7-a823-2fe7b0e2280e for client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.033 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 1c8bbd41-5c52-4bd7-a823-2fe7b0e2280e to client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.033 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 15:02:40.033 INFO [standby-run-42] StandbyClientSyncExecution.java:196 Copying data segment 1c8bbd41-5c52-4bd7-a823-2fe7b0e2280e from primary 15:02:40.033 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 420f258f-b6b4-4767-858d-0f47bf468a42 for segment 1c8bbd41-5c52-4bd7-a823-2fe7b0e2280e 15:02:40.033 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:02:40.033 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.033 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 1c8bbd41-5c52-4bd7-a823-2fe7b0e2280e for client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.033 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 192 sent to client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.033 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 1c8bbd41-5c52-4bd7-a823-2fe7b0e2280e to client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.033 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:02:40.033 INFO [standby-run-42] StandbyClientSyncExecution.java:196 Copying data segment 9cd0a741-268d-4bc0-a351-4e315ff45bd4 from primary 15:02:40.033 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 420f258f-b6b4-4767-858d-0f47bf468a42 for segment 9cd0a741-268d-4bc0-a351-4e315ff45bd4 15:02:40.033 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 15:02:40.033 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get segment' received from client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.033 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 9cd0a741-268d-4bc0-a351-4e315ff45bd4 for client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.033 DEBUG [primary-2] CommunicationObserver.java:98 Segment with size 448 sent to client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.033 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 9cd0a741-268d-4bc0-a351-4e315ff45bd4 to client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.033 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 15:02:40.033 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client 420f258f-b6b4-4767-858d-0f47bf468a42 for blob bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880 15:02:40.033 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 15:02:40.033 DEBUG [primary-2] CommunicationObserver.java:79 Message 'get blob id' received from client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.033 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880 for client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.033 DEBUG [primary-2] CommunicationObserver.java:105 Binary with size 5242880 sent to client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.033 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880 to client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.049 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880 to client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.049 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880 to client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.064 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:02:40.064 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880 15:02:40.064 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880.tmp 15:02:40.064 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880 to client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.080 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:02:40.080 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880 15:02:40.080 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880.tmp 15:02:40.080 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880 to client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.096 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880 to client 420f258f-b6b4-4767-858d-0f47bf468a42 15:02:40.096 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:02:40.096 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880 15:02:40.096 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880.tmp 15:02:40.118 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:02:40.118 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880 15:02:40.118 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880.tmp 15:02:40.118 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 15:02:40.118 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880 15:02:40.118 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880.tmp 15:02:40.133 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880 15:02:40.180 DEBUG [standby-run-42] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\bf489e4974f2067c895ccc28f8c8c6b416ff4d3b1fa1fde5a3b7fc2dba9186d1#5242880.tmp 15:02:40.333 INFO [standby-run-42] StandbyClientSyncExecution.java:82 updated head state successfully: true in 300ms. 15:02:40.333 DEBUG [standby-run-42] StandbyClient.java:157 Channel closed 15:02:42.665 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:02:42.665 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:02:42.665 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:02:42.665 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:02:42.665 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:02:42.696 INFO [main] FileStore.java:471 TarMK closed: target\junit8277413677521668728\junit6747363152698468464 15:02:42.734 INFO [main] FileStore.java:471 TarMK closed: target\junit8277413677521668728\junit6329151603887954162 15:02:42.750 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit953665662749889460\junit342384424426468879, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:02:42.750 INFO [main] FileStore.java:240 TarMK opened: target\junit953665662749889460\junit342384424426468879 (mmap=false) 15:02:42.750 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit953665662749889460\junit4866160738538944919, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 15:02:42.765 INFO [main] FileStore.java:240 TarMK opened: target\junit953665662749889460\junit4866160738538944919 (mmap=false) 15:02:42.796 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 15:02:42.796 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 15:03:07.882 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x91e512ea]' 15:03:07.882 DEBUG [main] StandbyServer.java:249 Binding was successful 15:03:07.882 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 69e99693-173a-4eb4-9333-16ae43901fc7 for current head 15:03:07.882 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:59449 is allowed 15:03:07.882 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 15:03:07.898 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get head' received from client 69e99693-173a-4eb4-9333-16ae43901fc7 15:03:07.898 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 69e99693-173a-4eb4-9333-16ae43901fc7 15:03:08.024 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 3f01cacd-b30b-49e7-a966-164c38890dc0.00000005 to client 69e99693-173a-4eb4-9333-16ae43901fc7 15:03:08.024 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 15:03:08.024 DEBUG [standby-run-43] StandbyClientSyncExecution.java:103 Found missing segment 3f01cacd-b30b-49e7-a966-164c38890dc0 15:03:08.025 DEBUG [standby-run-43] StandbyClientSyncExecution.java:124 Inspecting segment 3f01cacd-b30b-49e7-a966-164c38890dc0 15:03:08.025 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 69e99693-173a-4eb4-9333-16ae43901fc7 for references of segment 3f01cacd-b30b-49e7-a966-164c38890dc0 15:03:08.025 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 15:03:08.025 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 3f01cacd-b30b-49e7-a966-164c38890dc0 for client 69e99693-173a-4eb4-9333-16ae43901fc7 15:03:08.025 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 3f01cacd-b30b-49e7-a966-164c38890dc0 to client 69e99693-173a-4eb4-9333-16ae43901fc7 15:03:08.026 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 15:03:08.026 INFO [standby-run-43] StandbyClientSyncExecution.java:196 Copying data segment 3f01cacd-b30b-49e7-a966-164c38890dc0 from primary 15:03:08.026 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 69e99693-173a-4eb4-9333-16ae43901fc7 for segment 3f01cacd-b30b-49e7-a966-164c38890dc0 15:03:08.026 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 15:03:08.026 DEBUG [primary-1] CommunicationObserver.java:79 Message 'get segment' received from client 69e99693-173a-4eb4-9333-16ae43901fc7 15:03:08.027 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 3f01cacd-b30b-49e7-a966-164c38890dc0 for client 69e99693-173a-4eb4-9333-16ae43901fc7 15:03:08.027 DEBUG [primary-1] CommunicationObserver.java:98 Segment with size 192 sent to client 69e99693-173a-4eb4-9333-16ae43901fc7 15:03:08.027 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 3f01cacd-b30b-49e7-a966-164c38890dc0 to client 69e99693-173a-4eb4-9333-16ae43901fc7 15:03:08.028 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 15:03:08.137 INFO [standby-run-43] StandbyClientSyncExecution.java:82 updated head state successfully: true in 113ms. 15:03:08.152 DEBUG [standby-run-43] StandbyClient.java:157 Channel closed 15:03:10.370 DEBUG [main] StandbyClientSync.java:277 Group shut down 15:03:10.370 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:03:10.370 DEBUG [main] StandbyServer.java:220 Channel disconnected 15:03:10.370 DEBUG [main] StandbyServer.java:231 Boss group shut down 15:03:10.370 DEBUG [main] StandbyServer.java:237 Worker group shut down 15:03:10.455 INFO [main] FileStore.java:471 TarMK closed: target\junit953665662749889460\junit4866160738538944919 15:03:10.502 INFO [main] FileStore.java:471 TarMK closed: target\junit953665662749889460\junit342384424426468879