19:14:42.294 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1309469511819264434\junit4499592514908805288, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:14:42.310 INFO [main] FileStore.java:240 TarMK opened: target\junit1309469511819264434\junit4499592514908805288 (mmap=false) 19:14:42.410 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1309469511819264434\junit2485808571782511255, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:14:42.427 INFO [main] FileStore.java:240 TarMK opened: target\junit1309469511819264434\junit2485808571782511255 (mmap=false) 19:14:42.448 INFO [main] DataStoreTestBase.java:127 Test begin: testSyncFailingDueToTooShortTimeout 19:14:42.464 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 19:14:42.464 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 19:14:42.633 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xa1329342]' 19:14:42.633 DEBUG [main] StandbyServer.java:249 Binding was successful 19:14:43.049 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 446a6247-aa71-4574-a586-9ec9163f6cd9 for current head 19:14:43.049 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:63264 is allowed 19:14:43.049 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 19:14:43.049 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.049 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.049 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 37ede41f-b797-4a9d-a5f0-c4062cd4051f.0000000c to client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.049 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 19:14:43.049 DEBUG [standby-run-23] StandbyClientSyncExecution.java:103 Found missing segment 37ede41f-b797-4a9d-a5f0-c4062cd4051f 19:14:43.049 DEBUG [standby-run-23] StandbyClientSyncExecution.java:124 Inspecting segment 37ede41f-b797-4a9d-a5f0-c4062cd4051f 19:14:43.049 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 446a6247-aa71-4574-a586-9ec9163f6cd9 for references of segment 37ede41f-b797-4a9d-a5f0-c4062cd4051f 19:14:43.049 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 19:14:43.049 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 37ede41f-b797-4a9d-a5f0-c4062cd4051f for client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.049 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 37ede41f-b797-4a9d-a5f0-c4062cd4051f to client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.049 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:14:43.049 DEBUG [standby-run-23] StandbyClientSyncExecution.java:184 Found reference from 37ede41f-b797-4a9d-a5f0-c4062cd4051f to 8a5ce8fe-e4c2-409c-a76b-01aff38b7ca0 19:14:43.049 DEBUG [standby-run-23] StandbyClientSyncExecution.java:124 Inspecting segment 8a5ce8fe-e4c2-409c-a76b-01aff38b7ca0 19:14:43.049 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 446a6247-aa71-4574-a586-9ec9163f6cd9 for references of segment 8a5ce8fe-e4c2-409c-a76b-01aff38b7ca0 19:14:43.049 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 19:14:43.049 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 8a5ce8fe-e4c2-409c-a76b-01aff38b7ca0 for client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.049 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 8a5ce8fe-e4c2-409c-a76b-01aff38b7ca0 to client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.049 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:14:43.049 INFO [standby-run-23] StandbyClientSyncExecution.java:196 Copying data segment 8a5ce8fe-e4c2-409c-a76b-01aff38b7ca0 from primary 19:14:43.049 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 446a6247-aa71-4574-a586-9ec9163f6cd9 for segment 8a5ce8fe-e4c2-409c-a76b-01aff38b7ca0 19:14:43.049 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 19:14:43.049 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.049 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 8a5ce8fe-e4c2-409c-a76b-01aff38b7ca0 for client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.049 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.049 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 8a5ce8fe-e4c2-409c-a76b-01aff38b7ca0 to client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.049 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:14:43.049 INFO [standby-run-23] StandbyClientSyncExecution.java:196 Copying data segment 37ede41f-b797-4a9d-a5f0-c4062cd4051f from primary 19:14:43.049 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 446a6247-aa71-4574-a586-9ec9163f6cd9 for segment 37ede41f-b797-4a9d-a5f0-c4062cd4051f 19:14:43.049 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 19:14:43.049 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.049 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 37ede41f-b797-4a9d-a5f0-c4062cd4051f for client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.064 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.064 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 37ede41f-b797-4a9d-a5f0-c4062cd4051f to client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.064 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:14:43.064 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 446a6247-aa71-4574-a586-9ec9163f6cd9 for blob c0721a21f9ffcd426582735326d2b558e58c95b2ccc5abe9ab525786cabc9ccd#5242880 19:14:43.064 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 19:14:43.064 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get blob id' received from client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.064 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob c0721a21f9ffcd426582735326d2b558e58c95b2ccc5abe9ab525786cabc9ccd#5242880 for client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.064 DEBUG [primary-1] CommunicationObserver.java:130 Binary with size 5242880 sent to client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.064 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob c0721a21f9ffcd426582735326d2b558e58c95b2ccc5abe9ab525786cabc9ccd#5242880 to client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.080 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob c0721a21f9ffcd426582735326d2b558e58c95b2ccc5abe9ab525786cabc9ccd#5242880 to client 446a6247-aa71-4574-a586-9ec9163f6cd9 19:14:43.128 DEBUG [standby-run-23] StandbyClient.java:157 Channel closed 19:14:43.129 ERROR [standby-run-23] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to load remote blob c0721a21f9ffcd426582735326d2b558e58c95b2ccc5abe9ab525786cabc9ccd#5242880 at /root/server/#testBlob in 60ms. Please increase the timeout and try again. at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchAndStoreBlob(StandbyDiff.java:208) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:196) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:181) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:169) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeAdded(StandbyDiff.java:94) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeAdded(CancelableDiff.java:76) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:562) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:566) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:101) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.ExternalPrivateStoreIT.testSyncFailingDueToTooShortTimeout(ExternalPrivateStoreIT.java:86) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 19:14:45.380 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:14:45.380 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:14:45.380 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:14:45.380 DEBUG [main] StandbyServer.java:231 Boss group shut down 19:14:45.380 DEBUG [main] StandbyServer.java:237 Worker group shut down 19:14:45.380 INFO [main] DataStoreTestBase.java:132 Test end: testSyncFailingDueToTooShortTimeout 19:14:45.780 INFO [main] FileStore.java:466 TarMK closed: target\junit1309469511819264434\junit2485808571782511255 19:14:45.811 INFO [main] FileStore.java:466 TarMK closed: target\junit1309469511819264434\junit4499592514908805288 19:14:45.832 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7734990579563089546\junit161780084215743706, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:14:45.833 INFO [main] FileStore.java:240 TarMK opened: target\junit7734990579563089546\junit161780084215743706 (mmap=false) 19:14:45.833 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7734990579563089546\junit1788014097814363243, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:14:45.833 INFO [main] FileStore.java:240 TarMK opened: target\junit7734990579563089546\junit1788014097814363243 (mmap=false) 19:14:45.833 INFO [main] DataStoreTestBase.java:127 Test begin: testResilientSync 19:14:45.833 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 19:14:45.833 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 19:14:45.980 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xf1b281f2]' 19:14:45.980 DEBUG [main] StandbyServer.java:249 Binding was successful 19:14:45.980 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client d327cf42-48f6-4e10-87c3-c8824ea79183 for current head 19:14:45.980 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:63304 is allowed 19:14:45.980 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 19:14:45.980 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client d327cf42-48f6-4e10-87c3-c8824ea79183 19:14:45.980 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client d327cf42-48f6-4e10-87c3-c8824ea79183 19:14:49.991 ERROR [standby-run-24] StandbyClientSyncExecution.java:67 Unable to fetch remote head 19:14:49.991 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client d327cf42-48f6-4e10-87c3-c8824ea79183 io.netty.handler.timeout.ReadTimeoutException: null 19:14:49.991 DEBUG [standby-run-24] StandbyClient.java:157 Channel closed 19:14:51.259 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 8942514e-92bd-47ba-a120-a958c42d4204.0000000c to client d327cf42-48f6-4e10-87c3-c8824ea79183 19:14:51.259 WARN [primary-1] ExceptionHandler.java:31 Exception caught on the server java.io.IOException: Eine bestehende Verbindung wurde softwaregesteuert durch den Hostcomputer abgebrochen at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[na:1.8.0_151] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) ~[na:1.8.0_151] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_151] at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_151] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_151] at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288) ~[netty-buffer-4.1.14.Final.jar:4.1.14.Final] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106) ~[netty-buffer-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:372) ~[netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) ~[netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 19:14:52.199 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:14:52.199 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:14:52.199 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:14:52.199 DEBUG [main] StandbyServer.java:231 Boss group shut down 19:14:52.199 DEBUG [main] StandbyServer.java:237 Worker group shut down 19:14:52.268 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x9f42adc3]' 19:14:52.268 DEBUG [main] StandbyServer.java:249 Binding was successful 19:14:52.284 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client e918ab27-4c65-4029-8f65-0d58ef47f253 for current head 19:14:52.284 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:63345 is allowed 19:14:52.284 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 19:14:52.284 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.284 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.284 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 8942514e-92bd-47ba-a120-a958c42d4204.0000000c to client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.284 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 19:14:52.284 DEBUG [standby-run-25] StandbyClientSyncExecution.java:103 Found missing segment 8942514e-92bd-47ba-a120-a958c42d4204 19:14:52.284 DEBUG [standby-run-25] StandbyClientSyncExecution.java:124 Inspecting segment 8942514e-92bd-47ba-a120-a958c42d4204 19:14:52.284 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client e918ab27-4c65-4029-8f65-0d58ef47f253 for references of segment 8942514e-92bd-47ba-a120-a958c42d4204 19:14:52.284 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 19:14:52.284 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 8942514e-92bd-47ba-a120-a958c42d4204 for client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.284 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 8942514e-92bd-47ba-a120-a958c42d4204 to client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.284 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:14:52.284 DEBUG [standby-run-25] StandbyClientSyncExecution.java:184 Found reference from 8942514e-92bd-47ba-a120-a958c42d4204 to aa05d4cb-32a6-44f0-a4a6-fb4afdc9b75d 19:14:52.284 DEBUG [standby-run-25] StandbyClientSyncExecution.java:124 Inspecting segment aa05d4cb-32a6-44f0-a4a6-fb4afdc9b75d 19:14:52.284 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client e918ab27-4c65-4029-8f65-0d58ef47f253 for references of segment aa05d4cb-32a6-44f0-a4a6-fb4afdc9b75d 19:14:52.284 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 19:14:52.284 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment aa05d4cb-32a6-44f0-a4a6-fb4afdc9b75d for client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.284 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment aa05d4cb-32a6-44f0-a4a6-fb4afdc9b75d to client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.284 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:14:52.284 INFO [standby-run-25] StandbyClientSyncExecution.java:196 Copying data segment aa05d4cb-32a6-44f0-a4a6-fb4afdc9b75d from primary 19:14:52.284 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client e918ab27-4c65-4029-8f65-0d58ef47f253 for segment aa05d4cb-32a6-44f0-a4a6-fb4afdc9b75d 19:14:52.284 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 19:14:52.284 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.284 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment aa05d4cb-32a6-44f0-a4a6-fb4afdc9b75d for client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.284 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.284 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment aa05d4cb-32a6-44f0-a4a6-fb4afdc9b75d to client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.284 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:14:52.284 INFO [standby-run-25] StandbyClientSyncExecution.java:196 Copying data segment 8942514e-92bd-47ba-a120-a958c42d4204 from primary 19:14:52.284 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client e918ab27-4c65-4029-8f65-0d58ef47f253 for segment 8942514e-92bd-47ba-a120-a958c42d4204 19:14:52.284 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 19:14:52.284 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.284 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 8942514e-92bd-47ba-a120-a958c42d4204 for client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.284 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.284 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 8942514e-92bd-47ba-a120-a958c42d4204 to client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.284 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:14:52.299 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client e918ab27-4c65-4029-8f65-0d58ef47f253 for blob 399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880 19:14:52.299 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 19:14:52.299 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get blob id' received from client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.299 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob 399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880 for client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.299 DEBUG [primary-1] CommunicationObserver.java:130 Binary with size 5242880 sent to client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.299 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob 399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880 to client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.299 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880 to client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.349 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:14:52.350 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880 19:14:52.351 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880 to client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.353 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880.tmp 19:14:52.384 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:14:52.384 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880 19:14:52.384 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880 to client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.384 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880.tmp 19:14:52.431 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:14:52.431 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880 19:14:52.431 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880.tmp 19:14:52.431 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880 to client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.468 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:14:52.468 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880 19:14:52.468 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880.tmp 19:14:52.484 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880 to client e918ab27-4c65-4029-8f65-0d58ef47f253 19:14:52.515 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:14:52.515 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880 19:14:52.515 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880.tmp 19:14:52.531 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob 399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880 19:14:52.584 DEBUG [standby-run-25] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\399d85537d1af5688d6c0a5993ca05c69ea2dc0fa639e7325195d06435427b08#5242880.tmp 19:14:52.754 INFO [standby-run-25] StandbyClientSyncExecution.java:82 updated head state successfully: true in 470ms. 19:14:52.754 DEBUG [standby-run-25] StandbyClient.java:157 Channel closed 19:14:54.975 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:14:54.975 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:14:54.975 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:14:54.975 DEBUG [main] StandbyServer.java:231 Boss group shut down 19:14:54.975 DEBUG [main] StandbyServer.java:237 Worker group shut down 19:14:54.975 INFO [main] DataStoreTestBase.java:132 Test end: testResilientSync 19:14:55.022 INFO [main] FileStore.java:466 TarMK closed: target\junit7734990579563089546\junit1788014097814363243 19:14:55.060 INFO [main] FileStore.java:466 TarMK closed: target\junit7734990579563089546\junit161780084215743706 19:14:55.091 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7703797204022092704\junit4741557673890241794, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:14:55.091 INFO [main] FileStore.java:240 TarMK opened: target\junit7703797204022092704\junit4741557673890241794 (mmap=false) 19:14:55.106 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7703797204022092704\junit4367727854538717485, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:14:55.106 INFO [main] FileStore.java:240 TarMK opened: target\junit7703797204022092704\junit4367727854538717485 (mmap=false) 19:14:55.106 INFO [main] DataStoreTestBase.java:127 Test begin: testProxySkippedBytes 19:14:55.106 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 19:14:55.106 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 19:14:55.461 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x9a5873d6]' 19:14:55.461 DEBUG [main] StandbyServer.java:249 Binding was successful 19:14:55.593 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 10a3adaf-f529-4838-9bac-d165d44083f0 for current head 19:14:55.624 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:63434 is allowed 19:14:55.624 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 19:14:55.624 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 10a3adaf-f529-4838-9bac-d165d44083f0 19:14:55.624 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 10a3adaf-f529-4838-9bac-d165d44083f0 19:14:55.624 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head b182e002-d49b-4494-a911-3b9cce45f754.0000000c to client 10a3adaf-f529-4838-9bac-d165d44083f0 19:14:55.624 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 19:14:55.624 DEBUG [standby-run-26] StandbyClientSyncExecution.java:103 Found missing segment b182e002-d49b-4494-a911-3b9cce45f754 19:14:55.624 DEBUG [standby-run-26] StandbyClientSyncExecution.java:124 Inspecting segment b182e002-d49b-4494-a911-3b9cce45f754 19:14:55.624 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 10a3adaf-f529-4838-9bac-d165d44083f0 for references of segment b182e002-d49b-4494-a911-3b9cce45f754 19:14:55.624 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 19:14:55.624 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment b182e002-d49b-4494-a911-3b9cce45f754 for client 10a3adaf-f529-4838-9bac-d165d44083f0 19:14:55.624 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment b182e002-d49b-4494-a911-3b9cce45f754 to client 10a3adaf-f529-4838-9bac-d165d44083f0 19:15:00.642 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 10a3adaf-f529-4838-9bac-d165d44083f0 io.netty.handler.timeout.ReadTimeoutException: null 19:15:00.642 DEBUG [standby-run-26] StandbyClient.java:157 Channel closed 19:15:00.642 ERROR [standby-run-26] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment b182e002-d49b-4494-a911-3b9cce45f754 from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:346) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxySkippedBytes(DataStoreTestBase.java:296) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 19:15:02.980 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:15:03.011 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client b1e9856f-f63f-4750-bb0a-b09272595e7d for current head 19:15:03.011 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:63459 is allowed 19:15:03.011 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 19:15:03.011 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.011 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.011 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head b182e002-d49b-4494-a911-3b9cce45f754.0000000c to client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.011 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 19:15:03.011 DEBUG [standby-run-27] StandbyClientSyncExecution.java:103 Found missing segment b182e002-d49b-4494-a911-3b9cce45f754 19:15:03.011 DEBUG [standby-run-27] StandbyClientSyncExecution.java:124 Inspecting segment b182e002-d49b-4494-a911-3b9cce45f754 19:15:03.011 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client b1e9856f-f63f-4750-bb0a-b09272595e7d for references of segment b182e002-d49b-4494-a911-3b9cce45f754 19:15:03.011 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 19:15:03.011 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment b182e002-d49b-4494-a911-3b9cce45f754 for client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.011 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment b182e002-d49b-4494-a911-3b9cce45f754 to client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.011 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:15:03.011 DEBUG [standby-run-27] StandbyClientSyncExecution.java:184 Found reference from b182e002-d49b-4494-a911-3b9cce45f754 to feb547a6-1918-4a56-a0da-c2c7964ca781 19:15:03.011 DEBUG [standby-run-27] StandbyClientSyncExecution.java:124 Inspecting segment feb547a6-1918-4a56-a0da-c2c7964ca781 19:15:03.011 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client b1e9856f-f63f-4750-bb0a-b09272595e7d for references of segment feb547a6-1918-4a56-a0da-c2c7964ca781 19:15:03.011 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 19:15:03.011 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment feb547a6-1918-4a56-a0da-c2c7964ca781 for client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.011 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment feb547a6-1918-4a56-a0da-c2c7964ca781 to client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.011 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:15:03.011 INFO [standby-run-27] StandbyClientSyncExecution.java:196 Copying data segment feb547a6-1918-4a56-a0da-c2c7964ca781 from primary 19:15:03.011 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client b1e9856f-f63f-4750-bb0a-b09272595e7d for segment feb547a6-1918-4a56-a0da-c2c7964ca781 19:15:03.011 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 19:15:03.011 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.026 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment feb547a6-1918-4a56-a0da-c2c7964ca781 for client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.026 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.026 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment feb547a6-1918-4a56-a0da-c2c7964ca781 to client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.026 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:03.026 INFO [standby-run-27] StandbyClientSyncExecution.java:196 Copying data segment b182e002-d49b-4494-a911-3b9cce45f754 from primary 19:15:03.026 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client b1e9856f-f63f-4750-bb0a-b09272595e7d for segment b182e002-d49b-4494-a911-3b9cce45f754 19:15:03.026 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 19:15:03.026 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.026 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment b182e002-d49b-4494-a911-3b9cce45f754 for client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.026 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.026 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment b182e002-d49b-4494-a911-3b9cce45f754 to client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.026 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:03.026 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client b1e9856f-f63f-4750-bb0a-b09272595e7d for blob 80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880 19:15:03.026 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 19:15:03.026 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.026 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880 for client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.026 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 5242880 sent to client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.026 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880 to client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.026 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880 to client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.062 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:03.062 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880 19:15:03.064 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880 to client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.064 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880.tmp 19:15:03.080 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:03.080 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880 19:15:03.095 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880.tmp 19:15:03.095 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880 to client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.127 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:03.127 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880 19:15:03.127 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880.tmp 19:15:03.127 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880 to client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.180 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:03.180 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880 19:15:03.180 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880.tmp 19:15:03.180 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880 to client b1e9856f-f63f-4750-bb0a-b09272595e7d 19:15:03.211 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:03.211 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880 19:15:03.227 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880.tmp 19:15:03.227 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob 80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880 19:15:03.280 DEBUG [standby-run-27] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\80a8b19e478f8e96d8e4cf90798e8ff7b36325858f6bfb462592c6f674ee3dfe#5242880.tmp 19:15:03.427 INFO [standby-run-27] StandbyClientSyncExecution.java:82 updated head state successfully: true in 416ms. 19:15:03.427 DEBUG [standby-run-27] StandbyClient.java:157 Channel closed 19:15:05.662 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:15:05.663 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:15:05.664 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:15:05.665 DEBUG [main] StandbyServer.java:231 Boss group shut down 19:15:05.667 DEBUG [main] StandbyServer.java:237 Worker group shut down 19:15:05.683 INFO [main] DataStoreTestBase.java:132 Test end: testProxySkippedBytes 19:15:05.783 INFO [main] FileStore.java:466 TarMK closed: target\junit7703797204022092704\junit4367727854538717485 19:15:05.830 INFO [main] FileStore.java:466 TarMK closed: target\junit7703797204022092704\junit4741557673890241794 19:15:05.846 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit4725644355014850683\junit1269296813679832103, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:15:05.846 INFO [main] FileStore.java:240 TarMK opened: target\junit4725644355014850683\junit1269296813679832103 (mmap=false) 19:15:05.846 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit4725644355014850683\junit4848555449599043757, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:15:05.864 INFO [main] FileStore.java:240 TarMK opened: target\junit4725644355014850683\junit4848555449599043757 (mmap=false) 19:15:05.866 INFO [main] DataStoreTestBase.java:127 Test begin: testSyncUpdatedBinaryProperty 19:15:05.866 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 19:15:05.866 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 19:15:05.930 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x873c0dbd]' 19:15:05.930 DEBUG [main] StandbyServer.java:249 Binding was successful 19:15:06.130 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 for current head 19:15:06.130 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:63499 is allowed 19:15:06.130 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 19:15:06.130 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.130 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.130 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 55d85d1f-8999-449e-a99d-e1a804cf79c5.0000000c to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.130 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 19:15:06.130 DEBUG [standby-run-28] StandbyClientSyncExecution.java:103 Found missing segment 55d85d1f-8999-449e-a99d-e1a804cf79c5 19:15:06.130 DEBUG [standby-run-28] StandbyClientSyncExecution.java:124 Inspecting segment 55d85d1f-8999-449e-a99d-e1a804cf79c5 19:15:06.130 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 for references of segment 55d85d1f-8999-449e-a99d-e1a804cf79c5 19:15:06.130 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 19:15:06.130 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 55d85d1f-8999-449e-a99d-e1a804cf79c5 for client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.130 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 55d85d1f-8999-449e-a99d-e1a804cf79c5 to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.130 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:15:06.130 DEBUG [standby-run-28] StandbyClientSyncExecution.java:184 Found reference from 55d85d1f-8999-449e-a99d-e1a804cf79c5 to 60d9f909-cdb1-430a-ada3-9a0eca54122a 19:15:06.130 DEBUG [standby-run-28] StandbyClientSyncExecution.java:124 Inspecting segment 60d9f909-cdb1-430a-ada3-9a0eca54122a 19:15:06.130 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 for references of segment 60d9f909-cdb1-430a-ada3-9a0eca54122a 19:15:06.130 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 19:15:06.130 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 60d9f909-cdb1-430a-ada3-9a0eca54122a for client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.130 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 60d9f909-cdb1-430a-ada3-9a0eca54122a to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.130 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:15:06.146 INFO [standby-run-28] StandbyClientSyncExecution.java:196 Copying data segment 60d9f909-cdb1-430a-ada3-9a0eca54122a from primary 19:15:06.146 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 for segment 60d9f909-cdb1-430a-ada3-9a0eca54122a 19:15:06.146 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 19:15:06.146 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.146 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 60d9f909-cdb1-430a-ada3-9a0eca54122a for client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.146 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.146 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 60d9f909-cdb1-430a-ada3-9a0eca54122a to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.146 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:06.146 INFO [standby-run-28] StandbyClientSyncExecution.java:196 Copying data segment 55d85d1f-8999-449e-a99d-e1a804cf79c5 from primary 19:15:06.146 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 for segment 55d85d1f-8999-449e-a99d-e1a804cf79c5 19:15:06.146 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 19:15:06.146 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.146 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 55d85d1f-8999-449e-a99d-e1a804cf79c5 for client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.146 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.146 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 55d85d1f-8999-449e-a99d-e1a804cf79c5 to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.146 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:06.146 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 for blob d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880 19:15:06.146 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 19:15:06.146 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get blob id' received from client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.146 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880 for client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.146 DEBUG [primary-1] CommunicationObserver.java:130 Binary with size 5242880 sent to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.146 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880 to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.164 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880 to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.216 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:06.216 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880 19:15:06.216 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880.tmp 19:15:06.216 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880 to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.269 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:06.269 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880 19:15:06.269 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880.tmp 19:15:06.269 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880 to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.316 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:06.316 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880 19:15:06.316 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880.tmp 19:15:06.316 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880 to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.369 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:06.369 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880 19:15:06.369 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880 to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.369 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880.tmp 19:15:06.416 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:06.416 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880 19:15:06.416 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880.tmp 19:15:06.432 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880 19:15:06.536 DEBUG [standby-run-28] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\d70345efaadae46c749c57557ccf8c49dccc65c3be4533ffe283b08ed3aa738c#5242880.tmp 19:15:06.649 INFO [standby-run-28] StandbyClientSyncExecution.java:82 updated head state successfully: true in 519ms. 19:15:06.649 DEBUG [standby-run-28] StandbyClient.java:157 Channel closed 19:15:06.880 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 for current head 19:15:06.880 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:63500 is allowed 19:15:06.880 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 19:15:06.880 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.880 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.880 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 1ecf6057-08c7-403e-ac43-3a7b70c7c4a9.00000006 to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.880 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 19:15:06.880 DEBUG [standby-run-29] StandbyClientSyncExecution.java:103 Found missing segment 1ecf6057-08c7-403e-ac43-3a7b70c7c4a9 19:15:06.880 DEBUG [standby-run-29] StandbyClientSyncExecution.java:124 Inspecting segment 1ecf6057-08c7-403e-ac43-3a7b70c7c4a9 19:15:06.880 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 for references of segment 1ecf6057-08c7-403e-ac43-3a7b70c7c4a9 19:15:06.880 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 19:15:06.880 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 1ecf6057-08c7-403e-ac43-3a7b70c7c4a9 for client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.880 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 1ecf6057-08c7-403e-ac43-3a7b70c7c4a9 to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.880 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 19:15:06.880 INFO [standby-run-29] StandbyClientSyncExecution.java:196 Copying data segment 1ecf6057-08c7-403e-ac43-3a7b70c7c4a9 from primary 19:15:06.880 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 for segment 1ecf6057-08c7-403e-ac43-3a7b70c7c4a9 19:15:06.880 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 19:15:06.880 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.880 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 1ecf6057-08c7-403e-ac43-3a7b70c7c4a9 for client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.880 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 352 sent to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.880 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 1ecf6057-08c7-403e-ac43-3a7b70c7c4a9 to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.880 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:06.896 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 for blob 7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880 19:15:06.896 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 19:15:06.896 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.896 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880 for client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.896 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 5242880 sent to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.896 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880 to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.896 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880 to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:06.965 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:06.965 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880 19:15:06.980 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880.tmp 19:15:06.980 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880 to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:07.012 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:07.012 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880 19:15:07.012 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880.tmp 19:15:07.012 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880 to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:07.049 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:07.049 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880 19:15:07.049 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880 to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:07.049 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880.tmp 19:15:07.081 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:07.081 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880 19:15:07.081 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880 to client f1566dc5-e49a-44e8-a91b-8ad9256bdb63 19:15:07.081 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880.tmp 19:15:07.112 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:07.112 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880 19:15:07.112 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880.tmp 19:15:07.112 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob 7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880 19:15:07.165 DEBUG [standby-run-29] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\7e3396d38b7da4a83b5af5ca4bdb4aaa7fe8cb9bcd63c47932049bd681d715f7#5242880.tmp 19:15:07.311 INFO [standby-run-29] StandbyClientSyncExecution.java:82 updated head state successfully: true in 431ms. 19:15:07.311 DEBUG [standby-run-29] StandbyClient.java:157 Channel closed 19:15:09.628 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:15:09.628 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:15:09.628 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:15:09.628 DEBUG [main] StandbyServer.java:231 Boss group shut down 19:15:09.628 DEBUG [main] StandbyServer.java:237 Worker group shut down 19:15:09.628 INFO [main] DataStoreTestBase.java:132 Test end: testSyncUpdatedBinaryProperty 19:15:09.682 INFO [main] FileStore.java:466 TarMK closed: target\junit4725644355014850683\junit4848555449599043757 19:15:09.713 INFO [main] FileStore.java:466 TarMK closed: target\junit4725644355014850683\junit1269296813679832103 19:15:09.751 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7185298826678647582\junit1854537455569016774, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:15:09.751 INFO [main] FileStore.java:240 TarMK opened: target\junit7185298826678647582\junit1854537455569016774 (mmap=false) 19:15:09.751 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7185298826678647582\junit5996939313199524154, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:15:09.751 INFO [main] FileStore.java:240 TarMK opened: target\junit7185298826678647582\junit5996939313199524154 (mmap=false) 19:15:09.751 INFO [main] DataStoreTestBase.java:127 Test begin: testSync 19:15:09.751 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 19:15:09.751 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 19:15:09.883 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x342ab767]' 19:15:09.883 DEBUG [main] StandbyServer.java:249 Binding was successful 19:15:09.998 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 75805f19-4207-40db-9a8a-efa359333816 for current head 19:15:09.998 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:63540 is allowed 19:15:09.998 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 19:15:09.998 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 75805f19-4207-40db-9a8a-efa359333816 19:15:09.998 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 75805f19-4207-40db-9a8a-efa359333816 19:15:09.998 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head b3ab7e87-8d82-4280-a36e-e60cdb2ef76d.0000000c to client 75805f19-4207-40db-9a8a-efa359333816 19:15:09.998 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 19:15:09.998 DEBUG [standby-run-30] StandbyClientSyncExecution.java:103 Found missing segment b3ab7e87-8d82-4280-a36e-e60cdb2ef76d 19:15:09.998 DEBUG [standby-run-30] StandbyClientSyncExecution.java:124 Inspecting segment b3ab7e87-8d82-4280-a36e-e60cdb2ef76d 19:15:09.998 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 75805f19-4207-40db-9a8a-efa359333816 for references of segment b3ab7e87-8d82-4280-a36e-e60cdb2ef76d 19:15:09.998 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 19:15:09.998 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment b3ab7e87-8d82-4280-a36e-e60cdb2ef76d for client 75805f19-4207-40db-9a8a-efa359333816 19:15:09.998 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment b3ab7e87-8d82-4280-a36e-e60cdb2ef76d to client 75805f19-4207-40db-9a8a-efa359333816 19:15:09.998 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:15:09.998 DEBUG [standby-run-30] StandbyClientSyncExecution.java:184 Found reference from b3ab7e87-8d82-4280-a36e-e60cdb2ef76d to 07717330-13d4-46b9-a5be-fcf08c73d4a6 19:15:09.998 DEBUG [standby-run-30] StandbyClientSyncExecution.java:124 Inspecting segment 07717330-13d4-46b9-a5be-fcf08c73d4a6 19:15:09.998 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 75805f19-4207-40db-9a8a-efa359333816 for references of segment 07717330-13d4-46b9-a5be-fcf08c73d4a6 19:15:09.998 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 19:15:09.998 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 07717330-13d4-46b9-a5be-fcf08c73d4a6 for client 75805f19-4207-40db-9a8a-efa359333816 19:15:09.998 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 07717330-13d4-46b9-a5be-fcf08c73d4a6 to client 75805f19-4207-40db-9a8a-efa359333816 19:15:09.998 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:15:09.998 INFO [standby-run-30] StandbyClientSyncExecution.java:196 Copying data segment 07717330-13d4-46b9-a5be-fcf08c73d4a6 from primary 19:15:09.998 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 75805f19-4207-40db-9a8a-efa359333816 for segment 07717330-13d4-46b9-a5be-fcf08c73d4a6 19:15:09.998 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 19:15:09.998 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 75805f19-4207-40db-9a8a-efa359333816 19:15:09.998 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 07717330-13d4-46b9-a5be-fcf08c73d4a6 for client 75805f19-4207-40db-9a8a-efa359333816 19:15:09.998 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client 75805f19-4207-40db-9a8a-efa359333816 19:15:09.998 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 07717330-13d4-46b9-a5be-fcf08c73d4a6 to client 75805f19-4207-40db-9a8a-efa359333816 19:15:09.998 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:09.998 INFO [standby-run-30] StandbyClientSyncExecution.java:196 Copying data segment b3ab7e87-8d82-4280-a36e-e60cdb2ef76d from primary 19:15:09.998 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 75805f19-4207-40db-9a8a-efa359333816 for segment b3ab7e87-8d82-4280-a36e-e60cdb2ef76d 19:15:09.998 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 19:15:09.998 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 75805f19-4207-40db-9a8a-efa359333816 19:15:09.998 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment b3ab7e87-8d82-4280-a36e-e60cdb2ef76d for client 75805f19-4207-40db-9a8a-efa359333816 19:15:09.998 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client 75805f19-4207-40db-9a8a-efa359333816 19:15:09.998 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment b3ab7e87-8d82-4280-a36e-e60cdb2ef76d to client 75805f19-4207-40db-9a8a-efa359333816 19:15:09.998 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:10.014 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 75805f19-4207-40db-9a8a-efa359333816 for blob 13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880 19:15:10.014 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 19:15:10.014 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get blob id' received from client 75805f19-4207-40db-9a8a-efa359333816 19:15:10.014 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob 13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880 for client 75805f19-4207-40db-9a8a-efa359333816 19:15:10.014 DEBUG [primary-1] CommunicationObserver.java:130 Binary with size 5242880 sent to client 75805f19-4207-40db-9a8a-efa359333816 19:15:10.014 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob 13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880 to client 75805f19-4207-40db-9a8a-efa359333816 19:15:10.014 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880 to client 75805f19-4207-40db-9a8a-efa359333816 19:15:10.052 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:10.052 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880 19:15:10.052 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880 to client 75805f19-4207-40db-9a8a-efa359333816 19:15:10.052 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880.tmp 19:15:10.098 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:10.098 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880 19:15:10.098 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880 to client 75805f19-4207-40db-9a8a-efa359333816 19:15:10.098 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880.tmp 19:15:10.130 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880 to client 75805f19-4207-40db-9a8a-efa359333816 19:15:10.152 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880 to client 75805f19-4207-40db-9a8a-efa359333816 19:15:10.167 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:10.167 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880 19:15:10.167 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880.tmp 19:15:10.214 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:10.214 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880 19:15:10.214 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880.tmp 19:15:10.247 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:10.247 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880 19:15:10.249 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880.tmp 19:15:10.268 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob 13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880 19:15:10.315 DEBUG [standby-run-30] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\13d2fd13e61d4bad328c67131629bdfaff161f92e4a944250fe29be8814280b5#5242880.tmp 19:15:10.451 INFO [standby-run-30] StandbyClientSyncExecution.java:82 updated head state successfully: true in 453ms. 19:15:10.452 DEBUG [standby-run-30] StandbyClient.java:157 Channel closed 19:15:12.669 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:15:12.669 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:15:12.669 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:15:12.669 DEBUG [main] StandbyServer.java:231 Boss group shut down 19:15:12.669 DEBUG [main] StandbyServer.java:237 Worker group shut down 19:15:12.700 INFO [main] DataStoreTestBase.java:132 Test end: testSync 19:15:12.732 INFO [main] FileStore.java:466 TarMK closed: target\junit7185298826678647582\junit5996939313199524154 19:15:12.754 INFO [main] FileStore.java:466 TarMK closed: target\junit7185298826678647582\junit1854537455569016774 19:15:12.785 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit976216090629610003\junit1182023647683034918, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:15:12.785 INFO [main] FileStore.java:240 TarMK opened: target\junit976216090629610003\junit1182023647683034918 (mmap=false) 19:15:12.954 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit976216090629610003\junit4545241311365889843, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:15:12.954 INFO [main] FileStore.java:240 TarMK opened: target\junit976216090629610003\junit4545241311365889843 (mmap=false) 19:15:12.954 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedIntermediateByteChange2 19:15:12.954 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 19:15:12.954 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 19:15:13.301 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x75fa19fc]' 19:15:13.317 DEBUG [main] StandbyServer.java:249 Binding was successful 19:15:13.417 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 76ce6305-6d9b-4a58-8795-72c1a43acf5a for current head 19:15:13.471 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:63632 is allowed 19:15:13.471 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 19:15:13.471 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.471 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.471 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 2e8c0e59-bb33-4b37-a4f7-ed601e313c36.0000000c to client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.471 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 19:15:13.471 DEBUG [standby-run-31] StandbyClientSyncExecution.java:103 Found missing segment 2e8c0e59-bb33-4b37-a4f7-ed601e313c36 19:15:13.471 DEBUG [standby-run-31] StandbyClientSyncExecution.java:124 Inspecting segment 2e8c0e59-bb33-4b37-a4f7-ed601e313c36 19:15:13.471 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 76ce6305-6d9b-4a58-8795-72c1a43acf5a for references of segment 2e8c0e59-bb33-4b37-a4f7-ed601e313c36 19:15:13.471 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 19:15:13.471 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 2e8c0e59-bb33-4b37-a4f7-ed601e313c36 for client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.471 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 2e8c0e59-bb33-4b37-a4f7-ed601e313c36 to client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.471 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:15:13.471 DEBUG [standby-run-31] StandbyClientSyncExecution.java:184 Found reference from 2e8c0e59-bb33-4b37-a4f7-ed601e313c36 to 8a76f6bc-e12a-40af-a4a7-c282c9e86adc 19:15:13.471 DEBUG [standby-run-31] StandbyClientSyncExecution.java:124 Inspecting segment 8a76f6bc-e12a-40af-a4a7-c282c9e86adc 19:15:13.471 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 76ce6305-6d9b-4a58-8795-72c1a43acf5a for references of segment 8a76f6bc-e12a-40af-a4a7-c282c9e86adc 19:15:13.471 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 19:15:13.471 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 8a76f6bc-e12a-40af-a4a7-c282c9e86adc for client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.471 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 8a76f6bc-e12a-40af-a4a7-c282c9e86adc to client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.471 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:15:13.471 INFO [standby-run-31] StandbyClientSyncExecution.java:196 Copying data segment 8a76f6bc-e12a-40af-a4a7-c282c9e86adc from primary 19:15:13.471 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 76ce6305-6d9b-4a58-8795-72c1a43acf5a for segment 8a76f6bc-e12a-40af-a4a7-c282c9e86adc 19:15:13.471 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 19:15:13.471 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.471 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 8a76f6bc-e12a-40af-a4a7-c282c9e86adc for client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.471 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.471 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 8a76f6bc-e12a-40af-a4a7-c282c9e86adc to client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.471 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:13.471 INFO [standby-run-31] StandbyClientSyncExecution.java:196 Copying data segment 2e8c0e59-bb33-4b37-a4f7-ed601e313c36 from primary 19:15:13.471 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 76ce6305-6d9b-4a58-8795-72c1a43acf5a for segment 2e8c0e59-bb33-4b37-a4f7-ed601e313c36 19:15:13.471 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 19:15:13.471 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.471 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 2e8c0e59-bb33-4b37-a4f7-ed601e313c36 for client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.471 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.471 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 2e8c0e59-bb33-4b37-a4f7-ed601e313c36 to client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.471 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:13.471 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 76ce6305-6d9b-4a58-8795-72c1a43acf5a for blob f8f4dd9275f54c2d927ae8969ec3672c0f7a3168f0aaafc931b2c20a28b93ffa#5242880 19:15:13.471 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 19:15:13.471 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get blob id' received from client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.471 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob f8f4dd9275f54c2d927ae8969ec3672c0f7a3168f0aaafc931b2c20a28b93ffa#5242880 for client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.471 DEBUG [primary-1] CommunicationObserver.java:130 Binary with size 5242880 sent to client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.471 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob f8f4dd9275f54c2d927ae8969ec3672c0f7a3168f0aaafc931b2c20a28b93ffa#5242880 to client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.486 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob f8f4dd9275f54c2d927ae8969ec3672c0f7a3168f0aaafc931b2c20a28b93ffa#5242880 to client 76ce6305-6d9b-4a58-8795-72c1a43acf5a 19:15:13.502 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 76ce6305-6d9b-4a58-8795-72c1a43acf5a io.netty.handler.codec.compression.DecompressionException: mismatching checksum: bc29525d (expected: fb6dc88b) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 19:15:18.488 DEBUG [standby-run-31] StandbyClient.java:157 Channel closed 19:15:18.488 ERROR [standby-run-31] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to load remote blob f8f4dd9275f54c2d927ae8969ec3672c0f7a3168f0aaafc931b2c20a28b93ffa#5242880 at /root/server/#testBlob in 5000ms. Please increase the timeout and try again. at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchAndStoreBlob(StandbyDiff.java:208) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:196) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:181) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:169) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeAdded(StandbyDiff.java:94) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeAdded(CancelableDiff.java:76) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:562) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:566) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:101) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:346) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByteChange2(DataStoreTestBase.java:326) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 19:15:20.708 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:15:20.940 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 2da74f8b-3bae-4c12-ac28-26c0d995f860 for current head 19:15:20.940 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:63656 is allowed 19:15:20.940 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 19:15:20.940 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 2da74f8b-3bae-4c12-ac28-26c0d995f860 19:15:20.940 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 2da74f8b-3bae-4c12-ac28-26c0d995f860 19:15:20.940 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 453b8638-eea5-43fc-a15e-edf72d93fbbc.00000006 to client 2da74f8b-3bae-4c12-ac28-26c0d995f860 19:15:20.940 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 19:15:20.940 DEBUG [standby-run-32] StandbyClientSyncExecution.java:103 Found missing segment 453b8638-eea5-43fc-a15e-edf72d93fbbc 19:15:20.940 DEBUG [standby-run-32] StandbyClientSyncExecution.java:124 Inspecting segment 453b8638-eea5-43fc-a15e-edf72d93fbbc 19:15:20.940 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 2da74f8b-3bae-4c12-ac28-26c0d995f860 for references of segment 453b8638-eea5-43fc-a15e-edf72d93fbbc 19:15:20.940 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 19:15:20.940 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 453b8638-eea5-43fc-a15e-edf72d93fbbc for client 2da74f8b-3bae-4c12-ac28-26c0d995f860 19:15:20.940 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 453b8638-eea5-43fc-a15e-edf72d93fbbc to client 2da74f8b-3bae-4c12-ac28-26c0d995f860 19:15:20.940 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:15:20.940 INFO [standby-run-32] StandbyClientSyncExecution.java:196 Copying data segment 453b8638-eea5-43fc-a15e-edf72d93fbbc from primary 19:15:20.940 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 2da74f8b-3bae-4c12-ac28-26c0d995f860 for segment 453b8638-eea5-43fc-a15e-edf72d93fbbc 19:15:20.940 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 19:15:20.940 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 2da74f8b-3bae-4c12-ac28-26c0d995f860 19:15:20.940 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 453b8638-eea5-43fc-a15e-edf72d93fbbc for client 2da74f8b-3bae-4c12-ac28-26c0d995f860 19:15:20.940 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 352 sent to client 2da74f8b-3bae-4c12-ac28-26c0d995f860 19:15:20.940 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 453b8638-eea5-43fc-a15e-edf72d93fbbc to client 2da74f8b-3bae-4c12-ac28-26c0d995f860 19:15:20.940 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:20.940 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 2da74f8b-3bae-4c12-ac28-26c0d995f860 for blob 576063601b90ac677a36c55321319f18529c90074de081b93023df52ab306876#2097152 19:15:20.940 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 19:15:20.940 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client 2da74f8b-3bae-4c12-ac28-26c0d995f860 19:15:20.940 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 576063601b90ac677a36c55321319f18529c90074de081b93023df52ab306876#2097152 for client 2da74f8b-3bae-4c12-ac28-26c0d995f860 19:15:20.940 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 2097152 sent to client 2da74f8b-3bae-4c12-ac28-26c0d995f860 19:15:20.940 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 576063601b90ac677a36c55321319f18529c90074de081b93023df52ab306876#2097152 to client 2da74f8b-3bae-4c12-ac28-26c0d995f860 19:15:20.940 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/2 of size 1048576 from blob 576063601b90ac677a36c55321319f18529c90074de081b93023df52ab306876#2097152 to client 2da74f8b-3bae-4c12-ac28-26c0d995f860 19:15:20.963 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:20.963 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/2 of size 1048576 from blob 576063601b90ac677a36c55321319f18529c90074de081b93023df52ab306876#2097152 19:15:20.963 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/2 of size 1048576 from blob 576063601b90ac677a36c55321319f18529c90074de081b93023df52ab306876#2097152 to client 2da74f8b-3bae-4c12-ac28-26c0d995f860 19:15:20.963 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\576063601b90ac677a36c55321319f18529c90074de081b93023df52ab306876#2097152.tmp 19:15:21.010 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:21.010 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/2 of size 1048576 from blob 576063601b90ac677a36c55321319f18529c90074de081b93023df52ab306876#2097152 19:15:21.010 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\576063601b90ac677a36c55321319f18529c90074de081b93023df52ab306876#2097152.tmp 19:15:21.010 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob 576063601b90ac677a36c55321319f18529c90074de081b93023df52ab306876#2097152 19:15:21.041 DEBUG [standby-run-32] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\576063601b90ac677a36c55321319f18529c90074de081b93023df52ab306876#2097152.tmp 19:15:21.163 INFO [standby-run-32] StandbyClientSyncExecution.java:82 updated head state successfully: true in 223ms. 19:15:21.163 DEBUG [standby-run-32] StandbyClient.java:157 Channel closed 19:15:23.417 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:15:23.417 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:15:23.417 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:15:23.417 DEBUG [main] StandbyServer.java:231 Boss group shut down 19:15:23.417 DEBUG [main] StandbyServer.java:237 Worker group shut down 19:15:23.433 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedIntermediateByteChange2 19:15:23.533 INFO [main] FileStore.java:466 TarMK closed: target\junit976216090629610003\junit4545241311365889843 19:15:23.571 INFO [main] FileStore.java:466 TarMK closed: target\junit976216090629610003\junit1182023647683034918 19:15:23.602 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7032203516963362931\junit5463955754909440231, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:15:23.602 INFO [main] FileStore.java:240 TarMK opened: target\junit7032203516963362931\junit5463955754909440231 (mmap=false) 19:15:23.618 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7032203516963362931\junit2102592879939172188, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:15:23.618 INFO [main] FileStore.java:240 TarMK opened: target\junit7032203516963362931\junit2102592879939172188 (mmap=false) 19:15:23.618 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedIntermediateByteChange 19:15:23.618 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 19:15:23.618 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 19:15:23.872 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x82a6b92d]' 19:15:23.872 DEBUG [main] StandbyServer.java:249 Binding was successful 19:15:23.972 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client bec1754c-bcad-46dc-a6c5-a57135eaa4f6 for current head 19:15:24.003 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:63745 is allowed 19:15:24.003 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 19:15:24.003 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client bec1754c-bcad-46dc-a6c5-a57135eaa4f6 19:15:24.003 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client bec1754c-bcad-46dc-a6c5-a57135eaa4f6 19:15:24.003 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 84c59fcb-ec08-4eef-a954-fb33c843fd6c.0000000c to client bec1754c-bcad-46dc-a6c5-a57135eaa4f6 19:15:24.003 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 19:15:24.003 DEBUG [standby-run-33] StandbyClientSyncExecution.java:103 Found missing segment 84c59fcb-ec08-4eef-a954-fb33c843fd6c 19:15:24.003 DEBUG [standby-run-33] StandbyClientSyncExecution.java:124 Inspecting segment 84c59fcb-ec08-4eef-a954-fb33c843fd6c 19:15:24.003 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client bec1754c-bcad-46dc-a6c5-a57135eaa4f6 for references of segment 84c59fcb-ec08-4eef-a954-fb33c843fd6c 19:15:24.003 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 19:15:24.003 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 84c59fcb-ec08-4eef-a954-fb33c843fd6c for client bec1754c-bcad-46dc-a6c5-a57135eaa4f6 19:15:24.003 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 84c59fcb-ec08-4eef-a954-fb33c843fd6c to client bec1754c-bcad-46dc-a6c5-a57135eaa4f6 19:15:24.003 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client bec1754c-bcad-46dc-a6c5-a57135eaa4f6 io.netty.handler.codec.compression.DecompressionException: mismatching checksum: 71abbf91 (expected: a282e6ca) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 19:15:29.022 DEBUG [standby-run-33] StandbyClient.java:157 Channel closed 19:15:29.022 ERROR [standby-run-33] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment 84c59fcb-ec08-4eef-a954-fb33c843fd6c from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:346) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByteChange(DataStoreTestBase.java:321) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 19:15:31.272 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:15:31.491 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e for current head 19:15:31.491 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:63769 is allowed 19:15:31.491 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 19:15:31.491 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.491 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.491 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head a117aaee-a251-4e24-a722-ce88c89ef9b2.00000006 to client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.491 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 19:15:31.491 DEBUG [standby-run-34] StandbyClientSyncExecution.java:103 Found missing segment a117aaee-a251-4e24-a722-ce88c89ef9b2 19:15:31.491 DEBUG [standby-run-34] StandbyClientSyncExecution.java:124 Inspecting segment a117aaee-a251-4e24-a722-ce88c89ef9b2 19:15:31.491 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e for references of segment a117aaee-a251-4e24-a722-ce88c89ef9b2 19:15:31.491 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 19:15:31.507 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment a117aaee-a251-4e24-a722-ce88c89ef9b2 for client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.507 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment a117aaee-a251-4e24-a722-ce88c89ef9b2 to client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.507 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:15:31.507 DEBUG [standby-run-34] StandbyClientSyncExecution.java:184 Found reference from a117aaee-a251-4e24-a722-ce88c89ef9b2 to 84c59fcb-ec08-4eef-a954-fb33c843fd6c 19:15:31.507 DEBUG [standby-run-34] StandbyClientSyncExecution.java:184 Found reference from a117aaee-a251-4e24-a722-ce88c89ef9b2 to a15f321f-ab51-4814-aafe-0a3a74a004cf 19:15:31.507 DEBUG [standby-run-34] StandbyClientSyncExecution.java:124 Inspecting segment 84c59fcb-ec08-4eef-a954-fb33c843fd6c 19:15:31.507 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e for references of segment 84c59fcb-ec08-4eef-a954-fb33c843fd6c 19:15:31.507 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 19:15:31.507 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 84c59fcb-ec08-4eef-a954-fb33c843fd6c for client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.507 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 84c59fcb-ec08-4eef-a954-fb33c843fd6c to client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.507 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:15:31.507 DEBUG [standby-run-34] StandbyClientSyncExecution.java:124 Inspecting segment a15f321f-ab51-4814-aafe-0a3a74a004cf 19:15:31.507 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e for references of segment a15f321f-ab51-4814-aafe-0a3a74a004cf 19:15:31.507 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 19:15:31.507 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment a15f321f-ab51-4814-aafe-0a3a74a004cf for client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.507 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment a15f321f-ab51-4814-aafe-0a3a74a004cf to client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.507 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:15:31.507 INFO [standby-run-34] StandbyClientSyncExecution.java:196 Copying data segment a15f321f-ab51-4814-aafe-0a3a74a004cf from primary 19:15:31.507 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e for segment a15f321f-ab51-4814-aafe-0a3a74a004cf 19:15:31.507 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 19:15:31.507 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.507 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment a15f321f-ab51-4814-aafe-0a3a74a004cf for client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.507 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.507 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment a15f321f-ab51-4814-aafe-0a3a74a004cf to client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.507 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:31.507 INFO [standby-run-34] StandbyClientSyncExecution.java:196 Copying data segment 84c59fcb-ec08-4eef-a954-fb33c843fd6c from primary 19:15:31.507 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e for segment 84c59fcb-ec08-4eef-a954-fb33c843fd6c 19:15:31.507 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 19:15:31.507 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.507 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 84c59fcb-ec08-4eef-a954-fb33c843fd6c for client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.507 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.507 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 84c59fcb-ec08-4eef-a954-fb33c843fd6c to client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.507 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:31.507 INFO [standby-run-34] StandbyClientSyncExecution.java:196 Copying data segment a117aaee-a251-4e24-a722-ce88c89ef9b2 from primary 19:15:31.507 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e for segment a117aaee-a251-4e24-a722-ce88c89ef9b2 19:15:31.507 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 19:15:31.507 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.507 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment a117aaee-a251-4e24-a722-ce88c89ef9b2 for client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.507 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 352 sent to client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.507 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment a117aaee-a251-4e24-a722-ce88c89ef9b2 to client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.507 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:31.507 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e for blob 60a33b71af133f51afad816ad6948928ceb96805d6156be7f227dc05acfe3156#2097152 19:15:31.522 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 19:15:31.522 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.522 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 60a33b71af133f51afad816ad6948928ceb96805d6156be7f227dc05acfe3156#2097152 for client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.522 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 2097152 sent to client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.522 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 60a33b71af133f51afad816ad6948928ceb96805d6156be7f227dc05acfe3156#2097152 to client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.522 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/2 of size 1048576 from blob 60a33b71af133f51afad816ad6948928ceb96805d6156be7f227dc05acfe3156#2097152 to client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.572 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:31.572 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/2 of size 1048576 from blob 60a33b71af133f51afad816ad6948928ceb96805d6156be7f227dc05acfe3156#2097152 19:15:31.575 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\60a33b71af133f51afad816ad6948928ceb96805d6156be7f227dc05acfe3156#2097152.tmp 19:15:31.575 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/2 of size 1048576 from blob 60a33b71af133f51afad816ad6948928ceb96805d6156be7f227dc05acfe3156#2097152 to client 6aa5ebd0-b9cd-4fb8-8790-fd324ea1fd3e 19:15:31.684 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:31.684 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/2 of size 1048576 from blob 60a33b71af133f51afad816ad6948928ceb96805d6156be7f227dc05acfe3156#2097152 19:15:31.687 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\60a33b71af133f51afad816ad6948928ceb96805d6156be7f227dc05acfe3156#2097152.tmp 19:15:31.693 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob 60a33b71af133f51afad816ad6948928ceb96805d6156be7f227dc05acfe3156#2097152 19:15:31.725 DEBUG [standby-run-34] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\60a33b71af133f51afad816ad6948928ceb96805d6156be7f227dc05acfe3156#2097152.tmp 19:15:31.818 INFO [standby-run-34] StandbyClientSyncExecution.java:82 updated head state successfully: true in 327ms. 19:15:31.818 DEBUG [standby-run-34] StandbyClient.java:157 Channel closed 19:15:34.059 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:15:34.060 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:15:34.061 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:15:34.063 DEBUG [main] StandbyServer.java:231 Boss group shut down 19:15:34.064 DEBUG [main] StandbyServer.java:237 Worker group shut down 19:15:34.080 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedIntermediateByteChange 19:15:34.126 INFO [main] FileStore.java:466 TarMK closed: target\junit7032203516963362931\junit2102592879939172188 19:15:34.160 INFO [main] FileStore.java:466 TarMK closed: target\junit7032203516963362931\junit5463955754909440231 19:15:34.180 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1924794716221323734\junit3720517375837020727, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:15:34.180 INFO [main] FileStore.java:240 TarMK opened: target\junit1924794716221323734\junit3720517375837020727 (mmap=false) 19:15:34.195 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1924794716221323734\junit2857776412795340797, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:15:34.195 INFO [main] FileStore.java:240 TarMK opened: target\junit1924794716221323734\junit2857776412795340797 (mmap=false) 19:15:34.195 INFO [main] DataStoreTestBase.java:127 Test begin: testProxySkippedBytesIntermediateChange 19:15:34.195 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 19:15:34.195 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 19:15:34.528 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xa01aeb64]' 19:15:34.528 DEBUG [main] StandbyServer.java:249 Binding was successful 19:15:34.666 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client e460239e-25dd-4f09-8291-dc113556c49c for current head 19:15:34.697 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:63860 is allowed 19:15:34.697 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 19:15:34.697 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client e460239e-25dd-4f09-8291-dc113556c49c 19:15:34.697 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client e460239e-25dd-4f09-8291-dc113556c49c 19:15:34.697 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head f602fa90-600f-4677-a475-8e7d4a70c4dd.0000000c to client e460239e-25dd-4f09-8291-dc113556c49c 19:15:34.697 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 19:15:34.713 DEBUG [standby-run-35] StandbyClientSyncExecution.java:103 Found missing segment f602fa90-600f-4677-a475-8e7d4a70c4dd 19:15:34.713 DEBUG [standby-run-35] StandbyClientSyncExecution.java:124 Inspecting segment f602fa90-600f-4677-a475-8e7d4a70c4dd 19:15:34.713 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client e460239e-25dd-4f09-8291-dc113556c49c for references of segment f602fa90-600f-4677-a475-8e7d4a70c4dd 19:15:34.713 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 19:15:34.713 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment f602fa90-600f-4677-a475-8e7d4a70c4dd for client e460239e-25dd-4f09-8291-dc113556c49c 19:15:34.713 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment f602fa90-600f-4677-a475-8e7d4a70c4dd to client e460239e-25dd-4f09-8291-dc113556c49c 19:15:39.721 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client e460239e-25dd-4f09-8291-dc113556c49c io.netty.handler.timeout.ReadTimeoutException: null 19:15:39.721 DEBUG [standby-run-35] StandbyClient.java:157 Channel closed 19:15:39.721 ERROR [standby-run-35] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment f602fa90-600f-4677-a475-8e7d4a70c4dd from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:346) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxySkippedBytesIntermediateChange(DataStoreTestBase.java:301) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 19:15:41.938 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:15:42.272 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 4c560294-c39b-4e83-8f2a-46cc2c60304c for current head 19:15:42.273 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:63885 is allowed 19:15:42.273 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 19:15:42.273 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.273 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.273 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head a7a25ccd-b96f-4da8-a0f6-c1d6da4cbeb9.00000006 to client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.274 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 19:15:42.274 DEBUG [standby-run-36] StandbyClientSyncExecution.java:103 Found missing segment a7a25ccd-b96f-4da8-a0f6-c1d6da4cbeb9 19:15:42.274 DEBUG [standby-run-36] StandbyClientSyncExecution.java:124 Inspecting segment a7a25ccd-b96f-4da8-a0f6-c1d6da4cbeb9 19:15:42.274 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 4c560294-c39b-4e83-8f2a-46cc2c60304c for references of segment a7a25ccd-b96f-4da8-a0f6-c1d6da4cbeb9 19:15:42.275 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 19:15:42.275 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment a7a25ccd-b96f-4da8-a0f6-c1d6da4cbeb9 for client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment a7a25ccd-b96f-4da8-a0f6-c1d6da4cbeb9 to client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:15:42.275 DEBUG [standby-run-36] StandbyClientSyncExecution.java:184 Found reference from a7a25ccd-b96f-4da8-a0f6-c1d6da4cbeb9 to f602fa90-600f-4677-a475-8e7d4a70c4dd 19:15:42.275 DEBUG [standby-run-36] StandbyClientSyncExecution.java:184 Found reference from a7a25ccd-b96f-4da8-a0f6-c1d6da4cbeb9 to bd74b679-d8ec-4303-abe6-24e5e5597f3b 19:15:42.275 DEBUG [standby-run-36] StandbyClientSyncExecution.java:124 Inspecting segment f602fa90-600f-4677-a475-8e7d4a70c4dd 19:15:42.275 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 4c560294-c39b-4e83-8f2a-46cc2c60304c for references of segment f602fa90-600f-4677-a475-8e7d4a70c4dd 19:15:42.275 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 19:15:42.275 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment f602fa90-600f-4677-a475-8e7d4a70c4dd for client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment f602fa90-600f-4677-a475-8e7d4a70c4dd to client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:15:42.275 DEBUG [standby-run-36] StandbyClientSyncExecution.java:124 Inspecting segment bd74b679-d8ec-4303-abe6-24e5e5597f3b 19:15:42.275 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 4c560294-c39b-4e83-8f2a-46cc2c60304c for references of segment bd74b679-d8ec-4303-abe6-24e5e5597f3b 19:15:42.275 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 19:15:42.275 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment bd74b679-d8ec-4303-abe6-24e5e5597f3b for client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment bd74b679-d8ec-4303-abe6-24e5e5597f3b to client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:15:42.275 INFO [standby-run-36] StandbyClientSyncExecution.java:196 Copying data segment bd74b679-d8ec-4303-abe6-24e5e5597f3b from primary 19:15:42.275 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 4c560294-c39b-4e83-8f2a-46cc2c60304c for segment bd74b679-d8ec-4303-abe6-24e5e5597f3b 19:15:42.275 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 19:15:42.275 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment bd74b679-d8ec-4303-abe6-24e5e5597f3b for client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment bd74b679-d8ec-4303-abe6-24e5e5597f3b to client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:42.275 INFO [standby-run-36] StandbyClientSyncExecution.java:196 Copying data segment f602fa90-600f-4677-a475-8e7d4a70c4dd from primary 19:15:42.275 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 4c560294-c39b-4e83-8f2a-46cc2c60304c for segment f602fa90-600f-4677-a475-8e7d4a70c4dd 19:15:42.275 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 19:15:42.275 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment f602fa90-600f-4677-a475-8e7d4a70c4dd for client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment f602fa90-600f-4677-a475-8e7d4a70c4dd to client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:42.275 INFO [standby-run-36] StandbyClientSyncExecution.java:196 Copying data segment a7a25ccd-b96f-4da8-a0f6-c1d6da4cbeb9 from primary 19:15:42.275 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 4c560294-c39b-4e83-8f2a-46cc2c60304c for segment a7a25ccd-b96f-4da8-a0f6-c1d6da4cbeb9 19:15:42.275 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 19:15:42.275 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment a7a25ccd-b96f-4da8-a0f6-c1d6da4cbeb9 for client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 352 sent to client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment a7a25ccd-b96f-4da8-a0f6-c1d6da4cbeb9 to client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:42.275 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 4c560294-c39b-4e83-8f2a-46cc2c60304c for blob cd6df3e2cf5532ed35f8a1d5fa3539c22dfdfe970ef3becb37cbf45500a7b88a#2097152 19:15:42.275 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 19:15:42.275 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob cd6df3e2cf5532ed35f8a1d5fa3539c22dfdfe970ef3becb37cbf45500a7b88a#2097152 for client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 2097152 sent to client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob cd6df3e2cf5532ed35f8a1d5fa3539c22dfdfe970ef3becb37cbf45500a7b88a#2097152 to client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.275 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/2 of size 1048576 from blob cd6df3e2cf5532ed35f8a1d5fa3539c22dfdfe970ef3becb37cbf45500a7b88a#2097152 to client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.307 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:42.307 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/2 of size 1048576 from blob cd6df3e2cf5532ed35f8a1d5fa3539c22dfdfe970ef3becb37cbf45500a7b88a#2097152 19:15:42.307 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/2 of size 1048576 from blob cd6df3e2cf5532ed35f8a1d5fa3539c22dfdfe970ef3becb37cbf45500a7b88a#2097152 to client 4c560294-c39b-4e83-8f2a-46cc2c60304c 19:15:42.307 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\cd6df3e2cf5532ed35f8a1d5fa3539c22dfdfe970ef3becb37cbf45500a7b88a#2097152.tmp 19:15:42.338 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:42.338 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/2 of size 1048576 from blob cd6df3e2cf5532ed35f8a1d5fa3539c22dfdfe970ef3becb37cbf45500a7b88a#2097152 19:15:42.338 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\cd6df3e2cf5532ed35f8a1d5fa3539c22dfdfe970ef3becb37cbf45500a7b88a#2097152.tmp 19:15:42.354 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob cd6df3e2cf5532ed35f8a1d5fa3539c22dfdfe970ef3becb37cbf45500a7b88a#2097152 19:15:42.376 DEBUG [standby-run-36] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\cd6df3e2cf5532ed35f8a1d5fa3539c22dfdfe970ef3becb37cbf45500a7b88a#2097152.tmp 19:15:42.538 INFO [standby-run-36] StandbyClientSyncExecution.java:82 updated head state successfully: true in 264ms. 19:15:42.554 DEBUG [standby-run-36] StandbyClient.java:157 Channel closed 19:15:44.760 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:15:44.760 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:15:44.760 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:15:44.760 DEBUG [main] StandbyServer.java:231 Boss group shut down 19:15:44.760 DEBUG [main] StandbyServer.java:237 Worker group shut down 19:15:44.760 INFO [main] DataStoreTestBase.java:132 Test end: testProxySkippedBytesIntermediateChange 19:15:44.798 INFO [main] FileStore.java:466 TarMK closed: target\junit1924794716221323734\junit2857776412795340797 19:15:44.829 INFO [main] FileStore.java:466 TarMK closed: target\junit1924794716221323734\junit3720517375837020727 19:15:44.845 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit796368362755019534\junit2581820545665533611, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:15:44.860 INFO [main] FileStore.java:240 TarMK opened: target\junit796368362755019534\junit2581820545665533611 (mmap=false) 19:15:44.860 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit796368362755019534\junit4707812462369695186, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:15:44.860 INFO [main] FileStore.java:240 TarMK opened: target\junit796368362755019534\junit4707812462369695186 (mmap=false) 19:15:44.860 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedIntermediateByte 19:15:44.860 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 19:15:44.860 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 19:15:45.248 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x6c6f8b08]' 19:15:45.248 DEBUG [main] StandbyServer.java:249 Binding was successful 19:15:45.348 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 21cff7a3-a99b-445e-b5c7-317395afc83c for current head 19:15:45.386 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:63974 is allowed 19:15:45.386 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 19:15:45.386 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 21cff7a3-a99b-445e-b5c7-317395afc83c 19:15:45.386 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 21cff7a3-a99b-445e-b5c7-317395afc83c 19:15:45.386 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head f98ed03a-a986-468d-ad80-3abf58d8a337.0000000c to client 21cff7a3-a99b-445e-b5c7-317395afc83c 19:15:45.386 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 19:15:45.386 DEBUG [standby-run-37] StandbyClientSyncExecution.java:103 Found missing segment f98ed03a-a986-468d-ad80-3abf58d8a337 19:15:45.386 DEBUG [standby-run-37] StandbyClientSyncExecution.java:124 Inspecting segment f98ed03a-a986-468d-ad80-3abf58d8a337 19:15:45.386 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 21cff7a3-a99b-445e-b5c7-317395afc83c for references of segment f98ed03a-a986-468d-ad80-3abf58d8a337 19:15:45.386 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 19:15:45.386 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment f98ed03a-a986-468d-ad80-3abf58d8a337 for client 21cff7a3-a99b-445e-b5c7-317395afc83c 19:15:45.402 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment f98ed03a-a986-468d-ad80-3abf58d8a337 to client 21cff7a3-a99b-445e-b5c7-317395afc83c 19:15:45.402 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 21cff7a3-a99b-445e-b5c7-317395afc83c io.netty.handler.codec.compression.DecompressionException: mismatching checksum: bbe51874 (expected: a281edaf) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 19:15:50.410 DEBUG [standby-run-37] StandbyClient.java:157 Channel closed 19:15:50.410 ERROR [standby-run-37] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment f98ed03a-a986-468d-ad80-3abf58d8a337 from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:346) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByte(DataStoreTestBase.java:311) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 19:15:52.642 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:15:52.726 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 22bce700-3048-4a85-b6bf-56cd9387be3b for current head 19:15:52.726 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:63998 is allowed 19:15:52.726 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 19:15:52.726 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.726 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.726 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head f98ed03a-a986-468d-ad80-3abf58d8a337.0000000c to client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.726 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 19:15:52.726 DEBUG [standby-run-38] StandbyClientSyncExecution.java:103 Found missing segment f98ed03a-a986-468d-ad80-3abf58d8a337 19:15:52.726 DEBUG [standby-run-38] StandbyClientSyncExecution.java:124 Inspecting segment f98ed03a-a986-468d-ad80-3abf58d8a337 19:15:52.741 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 22bce700-3048-4a85-b6bf-56cd9387be3b for references of segment f98ed03a-a986-468d-ad80-3abf58d8a337 19:15:52.741 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 19:15:52.741 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment f98ed03a-a986-468d-ad80-3abf58d8a337 for client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.741 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment f98ed03a-a986-468d-ad80-3abf58d8a337 to client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.741 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:15:52.741 DEBUG [standby-run-38] StandbyClientSyncExecution.java:184 Found reference from f98ed03a-a986-468d-ad80-3abf58d8a337 to 94c5a995-3b9b-4dea-aaad-5a906f3b164e 19:15:52.741 DEBUG [standby-run-38] StandbyClientSyncExecution.java:124 Inspecting segment 94c5a995-3b9b-4dea-aaad-5a906f3b164e 19:15:52.741 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 22bce700-3048-4a85-b6bf-56cd9387be3b for references of segment 94c5a995-3b9b-4dea-aaad-5a906f3b164e 19:15:52.741 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 19:15:52.741 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 94c5a995-3b9b-4dea-aaad-5a906f3b164e for client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.741 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 94c5a995-3b9b-4dea-aaad-5a906f3b164e to client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.741 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:15:52.741 INFO [standby-run-38] StandbyClientSyncExecution.java:196 Copying data segment 94c5a995-3b9b-4dea-aaad-5a906f3b164e from primary 19:15:52.741 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 22bce700-3048-4a85-b6bf-56cd9387be3b for segment 94c5a995-3b9b-4dea-aaad-5a906f3b164e 19:15:52.741 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 19:15:52.741 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.741 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 94c5a995-3b9b-4dea-aaad-5a906f3b164e for client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.741 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.741 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 94c5a995-3b9b-4dea-aaad-5a906f3b164e to client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.741 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:52.741 INFO [standby-run-38] StandbyClientSyncExecution.java:196 Copying data segment f98ed03a-a986-468d-ad80-3abf58d8a337 from primary 19:15:52.741 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 22bce700-3048-4a85-b6bf-56cd9387be3b for segment f98ed03a-a986-468d-ad80-3abf58d8a337 19:15:52.741 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 19:15:52.741 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.741 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment f98ed03a-a986-468d-ad80-3abf58d8a337 for client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.741 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.741 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment f98ed03a-a986-468d-ad80-3abf58d8a337 to client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.741 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:52.741 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 22bce700-3048-4a85-b6bf-56cd9387be3b for blob 86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880 19:15:52.741 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 19:15:52.741 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.741 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880 for client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.741 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 5242880 sent to client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.741 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880 to client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.757 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880 to client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.773 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:52.773 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880 19:15:52.773 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880.tmp 19:15:52.773 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880 to client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.796 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:52.796 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880 19:15:52.796 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880.tmp 19:15:52.796 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880 to client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.827 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:52.827 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880 19:15:52.827 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880.tmp 19:15:52.827 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880 to client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.858 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:52.858 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880 19:15:52.858 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880.tmp 19:15:52.858 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880 to client 22bce700-3048-4a85-b6bf-56cd9387be3b 19:15:52.896 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:15:52.896 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880 19:15:52.912 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880.tmp 19:15:52.912 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob 86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880 19:15:52.974 DEBUG [standby-run-38] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\86e788c1fc92e3c78984323b231556952cbe3d95df958dca3961141557302786#5242880.tmp 19:15:53.096 INFO [standby-run-38] StandbyClientSyncExecution.java:82 updated head state successfully: true in 370ms. 19:15:53.096 DEBUG [standby-run-38] StandbyClient.java:157 Channel closed 19:15:55.329 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:15:55.329 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:15:55.329 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:15:55.329 DEBUG [main] StandbyServer.java:231 Boss group shut down 19:15:55.329 DEBUG [main] StandbyServer.java:237 Worker group shut down 19:15:55.345 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedIntermediateByte 19:15:55.376 INFO [main] FileStore.java:466 TarMK closed: target\junit796368362755019534\junit4707812462369695186 19:15:55.414 INFO [main] FileStore.java:466 TarMK closed: target\junit796368362755019534\junit2581820545665533611 19:15:55.445 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit6125137717809497691\junit89711674559214333, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:15:55.445 INFO [main] FileStore.java:240 TarMK opened: target\junit6125137717809497691\junit89711674559214333 (mmap=false) 19:15:55.498 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit6125137717809497691\junit6619501392915780152, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:15:55.498 INFO [main] FileStore.java:240 TarMK opened: target\junit6125137717809497691\junit6619501392915780152 (mmap=false) 19:15:55.498 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedIntermediateByte2 19:15:55.498 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 19:15:55.498 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 19:15:55.761 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x95b737c1]' 19:15:55.761 DEBUG [main] StandbyServer.java:249 Binding was successful 19:15:55.861 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client e0c24e74-ec37-413e-b2a7-f239a5b776d9 for current head 19:15:55.899 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:64087 is allowed 19:15:55.899 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 19:15:55.899 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.914 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.914 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head ddb12ca9-3349-4ff4-a6d8-f79454bf1c74.0000000c to client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.914 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 19:15:55.914 DEBUG [standby-run-39] StandbyClientSyncExecution.java:103 Found missing segment ddb12ca9-3349-4ff4-a6d8-f79454bf1c74 19:15:55.914 DEBUG [standby-run-39] StandbyClientSyncExecution.java:124 Inspecting segment ddb12ca9-3349-4ff4-a6d8-f79454bf1c74 19:15:55.914 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client e0c24e74-ec37-413e-b2a7-f239a5b776d9 for references of segment ddb12ca9-3349-4ff4-a6d8-f79454bf1c74 19:15:55.914 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 19:15:55.914 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment ddb12ca9-3349-4ff4-a6d8-f79454bf1c74 for client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.914 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment ddb12ca9-3349-4ff4-a6d8-f79454bf1c74 to client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.914 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:15:55.914 DEBUG [standby-run-39] StandbyClientSyncExecution.java:184 Found reference from ddb12ca9-3349-4ff4-a6d8-f79454bf1c74 to 9203c4b3-3e65-4576-a62a-cbd2a439b75d 19:15:55.914 DEBUG [standby-run-39] StandbyClientSyncExecution.java:124 Inspecting segment 9203c4b3-3e65-4576-a62a-cbd2a439b75d 19:15:55.914 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client e0c24e74-ec37-413e-b2a7-f239a5b776d9 for references of segment 9203c4b3-3e65-4576-a62a-cbd2a439b75d 19:15:55.914 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 19:15:55.914 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 9203c4b3-3e65-4576-a62a-cbd2a439b75d for client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.914 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 9203c4b3-3e65-4576-a62a-cbd2a439b75d to client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.914 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:15:55.914 INFO [standby-run-39] StandbyClientSyncExecution.java:196 Copying data segment 9203c4b3-3e65-4576-a62a-cbd2a439b75d from primary 19:15:55.914 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client e0c24e74-ec37-413e-b2a7-f239a5b776d9 for segment 9203c4b3-3e65-4576-a62a-cbd2a439b75d 19:15:55.914 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 19:15:55.914 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.914 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 9203c4b3-3e65-4576-a62a-cbd2a439b75d for client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.914 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.914 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 9203c4b3-3e65-4576-a62a-cbd2a439b75d to client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.914 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:55.914 INFO [standby-run-39] StandbyClientSyncExecution.java:196 Copying data segment ddb12ca9-3349-4ff4-a6d8-f79454bf1c74 from primary 19:15:55.914 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client e0c24e74-ec37-413e-b2a7-f239a5b776d9 for segment ddb12ca9-3349-4ff4-a6d8-f79454bf1c74 19:15:55.914 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 19:15:55.914 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.914 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment ddb12ca9-3349-4ff4-a6d8-f79454bf1c74 for client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.914 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.914 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment ddb12ca9-3349-4ff4-a6d8-f79454bf1c74 to client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.914 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:15:55.914 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client e0c24e74-ec37-413e-b2a7-f239a5b776d9 for blob bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880 19:15:55.914 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 19:15:55.914 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get blob id' received from client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.914 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880 for client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.914 DEBUG [primary-1] CommunicationObserver.java:130 Binary with size 5242880 sent to client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.914 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880 to client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.930 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880 to client e0c24e74-ec37-413e-b2a7-f239a5b776d9 19:15:55.946 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client e0c24e74-ec37-413e-b2a7-f239a5b776d9 io.netty.handler.codec.compression.DecompressionException: mismatching checksum: a282d98c (expected: 372d5f99) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 19:16:00.936 DEBUG [standby-run-39] StandbyClient.java:157 Channel closed 19:16:00.936 ERROR [standby-run-39] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to load remote blob bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880 at /root/server/#testBlob in 5000ms. Please increase the timeout and try again. at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchAndStoreBlob(StandbyDiff.java:208) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:196) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:181) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:169) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeAdded(StandbyDiff.java:94) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeAdded(CancelableDiff.java:76) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:562) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:566) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:101) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:346) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByte2(DataStoreTestBase.java:316) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 19:16:03.153 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:16:03.206 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 302f8e3a-6a75-4565-a921-8314046fd19d for current head 19:16:03.206 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:64113 is allowed 19:16:03.206 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 19:16:03.206 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 302f8e3a-6a75-4565-a921-8314046fd19d 19:16:03.206 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 302f8e3a-6a75-4565-a921-8314046fd19d 19:16:03.206 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head ddb12ca9-3349-4ff4-a6d8-f79454bf1c74.0000000c to client 302f8e3a-6a75-4565-a921-8314046fd19d 19:16:03.206 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 19:16:03.206 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 302f8e3a-6a75-4565-a921-8314046fd19d for blob bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880 19:16:03.206 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 19:16:03.206 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client 302f8e3a-6a75-4565-a921-8314046fd19d 19:16:03.206 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880 for client 302f8e3a-6a75-4565-a921-8314046fd19d 19:16:03.206 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 5242880 sent to client 302f8e3a-6a75-4565-a921-8314046fd19d 19:16:03.206 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880 to client 302f8e3a-6a75-4565-a921-8314046fd19d 19:16:03.206 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880 to client 302f8e3a-6a75-4565-a921-8314046fd19d 19:16:03.238 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:16:03.238 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880 19:16:03.253 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880.tmp 19:16:03.253 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880 to client 302f8e3a-6a75-4565-a921-8314046fd19d 19:16:03.285 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:16:03.285 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880 19:16:03.285 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880.tmp 19:16:03.285 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880 to client 302f8e3a-6a75-4565-a921-8314046fd19d 19:16:03.322 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:16:03.322 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880 19:16:03.322 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880.tmp 19:16:03.322 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880 to client 302f8e3a-6a75-4565-a921-8314046fd19d 19:16:03.354 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:16:03.354 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880 19:16:03.354 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880 to client 302f8e3a-6a75-4565-a921-8314046fd19d 19:16:03.354 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880.tmp 19:16:03.369 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:16:03.385 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880 19:16:03.385 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880.tmp 19:16:03.385 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880 19:16:03.438 DEBUG [standby-run-40] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\bdf35d8343a8fa86b87ba146b101800c21627459d4b4ee45d358326da0e8ee56#5242880.tmp 19:16:03.585 INFO [standby-run-40] StandbyClientSyncExecution.java:82 updated head state successfully: true in 379ms. 19:16:03.600 DEBUG [standby-run-40] StandbyClient.java:157 Channel closed 19:16:05.842 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:16:05.842 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:16:05.842 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:16:05.842 DEBUG [main] StandbyServer.java:231 Boss group shut down 19:16:05.842 DEBUG [main] StandbyServer.java:237 Worker group shut down 19:16:05.873 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedIntermediateByte2 19:16:05.911 INFO [main] FileStore.java:466 TarMK closed: target\junit6125137717809497691\junit6619501392915780152 19:16:05.942 INFO [main] FileStore.java:466 TarMK closed: target\junit6125137717809497691\junit89711674559214333 19:16:05.973 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit8513754855467918375\junit6602314424654383577, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:16:05.973 INFO [main] FileStore.java:240 TarMK opened: target\junit8513754855467918375\junit6602314424654383577 (mmap=false) 19:16:05.973 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit8513754855467918375\junit4253978074730293660, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:16:05.973 INFO [main] FileStore.java:240 TarMK opened: target\junit8513754855467918375\junit4253978074730293660 (mmap=false) 19:16:05.989 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedStartByte 19:16:05.989 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 19:16:05.989 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 19:16:06.245 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xb1b353fa]' 19:16:06.247 DEBUG [main] StandbyServer.java:249 Binding was successful 19:16:06.387 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 90da30ef-a70e-433d-a7a9-baa9a9b28e22 for current head 19:16:06.438 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:64202 is allowed 19:16:06.438 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 19:16:06.439 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 90da30ef-a70e-433d-a7a9-baa9a9b28e22 19:16:06.439 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 90da30ef-a70e-433d-a7a9-baa9a9b28e22 19:16:06.439 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 2c57ba92-3027-4e0e-a2ef-42966c7dda88.0000000c to client 90da30ef-a70e-433d-a7a9-baa9a9b28e22 19:16:06.441 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 90da30ef-a70e-433d-a7a9-baa9a9b28e22 io.netty.handler.codec.compression.DecompressionException: Received COMPRESSED_DATA tag before STREAM_IDENTIFIER at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:162) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 19:16:11.388 ERROR [standby-run-41] StandbyClientSyncExecution.java:67 Unable to fetch remote head 19:16:11.389 DEBUG [standby-run-41] StandbyClient.java:157 Channel closed 19:16:13.640 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:16:13.693 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 57797e46-16e0-4155-a88e-3beb6fd7ef70 for current head 19:16:13.693 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:64226 is allowed 19:16:13.693 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 19:16:13.693 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.693 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.693 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 2c57ba92-3027-4e0e-a2ef-42966c7dda88.0000000c to client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.693 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 19:16:13.709 DEBUG [standby-run-42] StandbyClientSyncExecution.java:103 Found missing segment 2c57ba92-3027-4e0e-a2ef-42966c7dda88 19:16:13.709 DEBUG [standby-run-42] StandbyClientSyncExecution.java:124 Inspecting segment 2c57ba92-3027-4e0e-a2ef-42966c7dda88 19:16:13.709 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 57797e46-16e0-4155-a88e-3beb6fd7ef70 for references of segment 2c57ba92-3027-4e0e-a2ef-42966c7dda88 19:16:13.709 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 19:16:13.709 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 2c57ba92-3027-4e0e-a2ef-42966c7dda88 for client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.709 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 2c57ba92-3027-4e0e-a2ef-42966c7dda88 to client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.709 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:16:13.709 DEBUG [standby-run-42] StandbyClientSyncExecution.java:184 Found reference from 2c57ba92-3027-4e0e-a2ef-42966c7dda88 to b3783e4e-1be7-47e2-ade5-a3ab206c581c 19:16:13.709 DEBUG [standby-run-42] StandbyClientSyncExecution.java:124 Inspecting segment b3783e4e-1be7-47e2-ade5-a3ab206c581c 19:16:13.709 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 57797e46-16e0-4155-a88e-3beb6fd7ef70 for references of segment b3783e4e-1be7-47e2-ade5-a3ab206c581c 19:16:13.709 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 19:16:13.709 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment b3783e4e-1be7-47e2-ade5-a3ab206c581c for client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.709 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment b3783e4e-1be7-47e2-ade5-a3ab206c581c to client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.709 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:16:13.709 INFO [standby-run-42] StandbyClientSyncExecution.java:196 Copying data segment b3783e4e-1be7-47e2-ade5-a3ab206c581c from primary 19:16:13.709 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 57797e46-16e0-4155-a88e-3beb6fd7ef70 for segment b3783e4e-1be7-47e2-ade5-a3ab206c581c 19:16:13.709 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 19:16:13.709 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.709 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment b3783e4e-1be7-47e2-ade5-a3ab206c581c for client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.709 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.709 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment b3783e4e-1be7-47e2-ade5-a3ab206c581c to client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.709 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:16:13.709 INFO [standby-run-42] StandbyClientSyncExecution.java:196 Copying data segment 2c57ba92-3027-4e0e-a2ef-42966c7dda88 from primary 19:16:13.709 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 57797e46-16e0-4155-a88e-3beb6fd7ef70 for segment 2c57ba92-3027-4e0e-a2ef-42966c7dda88 19:16:13.709 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 19:16:13.709 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.709 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 2c57ba92-3027-4e0e-a2ef-42966c7dda88 for client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.709 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.709 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 2c57ba92-3027-4e0e-a2ef-42966c7dda88 to client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.709 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:16:13.709 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 57797e46-16e0-4155-a88e-3beb6fd7ef70 for blob 9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880 19:16:13.709 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 19:16:13.709 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.709 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880 for client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.709 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 5242880 sent to client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.709 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880 to client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.709 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880 to client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.756 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:16:13.756 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880 19:16:13.756 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880.tmp 19:16:13.756 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880 to client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.794 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:16:13.794 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880 19:16:13.794 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880 to client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.794 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880.tmp 19:16:13.826 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:16:13.826 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880 19:16:13.841 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880.tmp 19:16:13.841 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880 to client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.872 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:16:13.872 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880 19:16:13.872 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880.tmp 19:16:13.872 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880 to client 57797e46-16e0-4155-a88e-3beb6fd7ef70 19:16:13.911 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 19:16:13.911 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880 19:16:13.911 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880.tmp 19:16:13.911 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob 9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880 19:16:13.974 DEBUG [standby-run-42] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\9d9b71c70b297cf03ebe71e29b2cddbf379bf75e21b31e8a517b7b240685ca72#5242880.tmp 19:16:14.111 INFO [standby-run-42] StandbyClientSyncExecution.java:82 updated head state successfully: true in 418ms. 19:16:14.111 DEBUG [standby-run-42] StandbyClient.java:157 Channel closed 19:16:16.459 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:16:16.459 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:16:16.459 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:16:16.459 DEBUG [main] StandbyServer.java:231 Boss group shut down 19:16:16.459 DEBUG [main] StandbyServer.java:237 Worker group shut down 19:16:16.491 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedStartByte 19:16:16.527 INFO [main] FileStore.java:466 TarMK closed: target\junit8513754855467918375\junit4253978074730293660 19:16:16.558 INFO [main] FileStore.java:466 TarMK closed: target\junit8513754855467918375\junit6602314424654383577 19:16:16.596 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit475130445309960440\junit5561110862096072502, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:16:16.596 INFO [main] FileStore.java:240 TarMK opened: target\junit475130445309960440\junit5561110862096072502 (mmap=false) 19:16:16.627 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit475130445309960440\junit7217783835170550414, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 19:16:16.627 INFO [main] FileStore.java:240 TarMK opened: target\junit475130445309960440\junit7217783835170550414 (mmap=false) 19:16:16.627 INFO [main] DataStoreTestBase.java:127 Test begin: testSyncBigBlob 19:16:16.627 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 19:16:16.627 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 19:16:42.963 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xd24169d7]' 19:16:42.963 DEBUG [main] StandbyServer.java:249 Binding was successful 19:16:42.963 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 128e831b-6942-42f5-9dfd-e211bf2ff9e2 for current head 19:16:42.963 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:64297 is allowed 19:16:42.963 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 19:16:42.963 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 128e831b-6942-42f5-9dfd-e211bf2ff9e2 19:16:42.963 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 128e831b-6942-42f5-9dfd-e211bf2ff9e2 19:16:42.963 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head bb643543-98d6-40cf-a2bc-4bf86ae57042.00000005 to client 128e831b-6942-42f5-9dfd-e211bf2ff9e2 19:16:42.963 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 19:16:42.963 DEBUG [standby-run-43] StandbyClientSyncExecution.java:103 Found missing segment bb643543-98d6-40cf-a2bc-4bf86ae57042 19:16:42.963 DEBUG [standby-run-43] StandbyClientSyncExecution.java:124 Inspecting segment bb643543-98d6-40cf-a2bc-4bf86ae57042 19:16:42.963 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 128e831b-6942-42f5-9dfd-e211bf2ff9e2 for references of segment bb643543-98d6-40cf-a2bc-4bf86ae57042 19:16:42.963 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 19:16:42.963 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment bb643543-98d6-40cf-a2bc-4bf86ae57042 for client 128e831b-6942-42f5-9dfd-e211bf2ff9e2 19:16:42.994 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment bb643543-98d6-40cf-a2bc-4bf86ae57042 to client 128e831b-6942-42f5-9dfd-e211bf2ff9e2 19:16:42.994 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 19:16:42.994 INFO [standby-run-43] StandbyClientSyncExecution.java:196 Copying data segment bb643543-98d6-40cf-a2bc-4bf86ae57042 from primary 19:16:42.994 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 128e831b-6942-42f5-9dfd-e211bf2ff9e2 for segment bb643543-98d6-40cf-a2bc-4bf86ae57042 19:16:42.994 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 19:16:42.994 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 128e831b-6942-42f5-9dfd-e211bf2ff9e2 19:16:42.994 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment bb643543-98d6-40cf-a2bc-4bf86ae57042 for client 128e831b-6942-42f5-9dfd-e211bf2ff9e2 19:16:42.994 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client 128e831b-6942-42f5-9dfd-e211bf2ff9e2 19:16:42.994 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment bb643543-98d6-40cf-a2bc-4bf86ae57042 to client 128e831b-6942-42f5-9dfd-e211bf2ff9e2 19:16:42.994 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 19:16:43.394 INFO [standby-run-43] StandbyClientSyncExecution.java:82 updated head state successfully: true in 431ms. 19:16:43.394 DEBUG [standby-run-43] StandbyClient.java:157 Channel closed 19:16:45.625 DEBUG [main] StandbyClientSync.java:277 Group shut down 19:16:45.625 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:16:45.625 DEBUG [main] StandbyServer.java:220 Channel disconnected 19:16:45.625 DEBUG [main] StandbyServer.java:231 Boss group shut down 19:16:45.625 DEBUG [main] StandbyServer.java:237 Worker group shut down 19:16:45.625 INFO [main] DataStoreTestBase.java:132 Test end: testSyncBigBlob 19:16:45.725 INFO [main] FileStore.java:466 TarMK closed: target\junit475130445309960440\junit7217783835170550414 19:16:45.763 INFO [main] FileStore.java:466 TarMK closed: target\junit475130445309960440\junit5561110862096072502