16:42:03.701 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit5486475792104479439\junit7610614532259530189, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:42:03.701 INFO [main] FileStore.java:240 TarMK opened: target\junit5486475792104479439\junit7610614532259530189 (mmap=false) 16:42:03.701 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit5486475792104479439\junit3676281828139618322, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:42:03.717 INFO [main] FileStore.java:240 TarMK opened: target\junit5486475792104479439\junit3676281828139618322 (mmap=false) 16:42:03.717 INFO [main] DataStoreTestBase.java:127 Test begin: testSyncFailingDueToTooShortTimeout 16:42:03.717 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 16:42:03.717 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 16:42:03.833 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x79bcab25]' 16:42:03.833 DEBUG [main] StandbyServer.java:249 Binding was successful 16:42:03.917 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 1af42e6d-31ae-4caf-bc83-bac393fd0210 for current head 16:42:03.917 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61230 is allowed 16:42:03.917 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 16:42:03.917 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:03.917 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:03.917 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 971fd20e-fcf2-4aa3-a150-38b4ba75b52b.0000000c to client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:03.917 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 16:42:03.917 DEBUG [standby-run-23] StandbyClientSyncExecution.java:103 Found missing segment 971fd20e-fcf2-4aa3-a150-38b4ba75b52b 16:42:03.933 DEBUG [standby-run-23] StandbyClientSyncExecution.java:124 Inspecting segment 971fd20e-fcf2-4aa3-a150-38b4ba75b52b 16:42:03.933 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 1af42e6d-31ae-4caf-bc83-bac393fd0210 for references of segment 971fd20e-fcf2-4aa3-a150-38b4ba75b52b 16:42:03.933 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 16:42:03.933 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 971fd20e-fcf2-4aa3-a150-38b4ba75b52b for client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:03.933 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 971fd20e-fcf2-4aa3-a150-38b4ba75b52b to client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:03.933 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:42:03.933 DEBUG [standby-run-23] StandbyClientSyncExecution.java:184 Found reference from 971fd20e-fcf2-4aa3-a150-38b4ba75b52b to 9513ff82-7a51-42e4-a7b1-d034d60153cf 16:42:03.933 DEBUG [standby-run-23] StandbyClientSyncExecution.java:124 Inspecting segment 9513ff82-7a51-42e4-a7b1-d034d60153cf 16:42:03.933 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 1af42e6d-31ae-4caf-bc83-bac393fd0210 for references of segment 9513ff82-7a51-42e4-a7b1-d034d60153cf 16:42:03.933 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 16:42:03.933 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 9513ff82-7a51-42e4-a7b1-d034d60153cf for client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:03.933 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 9513ff82-7a51-42e4-a7b1-d034d60153cf to client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:03.933 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:42:03.933 INFO [standby-run-23] StandbyClientSyncExecution.java:196 Copying data segment 9513ff82-7a51-42e4-a7b1-d034d60153cf from primary 16:42:03.933 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 1af42e6d-31ae-4caf-bc83-bac393fd0210 for segment 9513ff82-7a51-42e4-a7b1-d034d60153cf 16:42:03.933 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 16:42:03.933 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:03.933 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 9513ff82-7a51-42e4-a7b1-d034d60153cf for client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:03.933 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:03.933 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 9513ff82-7a51-42e4-a7b1-d034d60153cf to client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:03.933 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:42:03.933 INFO [standby-run-23] StandbyClientSyncExecution.java:196 Copying data segment 971fd20e-fcf2-4aa3-a150-38b4ba75b52b from primary 16:42:03.933 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 1af42e6d-31ae-4caf-bc83-bac393fd0210 for segment 971fd20e-fcf2-4aa3-a150-38b4ba75b52b 16:42:03.933 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 16:42:03.933 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:03.933 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 971fd20e-fcf2-4aa3-a150-38b4ba75b52b for client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:03.933 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:03.933 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 971fd20e-fcf2-4aa3-a150-38b4ba75b52b to client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:03.933 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:42:03.949 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 1af42e6d-31ae-4caf-bc83-bac393fd0210 for blob a9df54dba0d08e22b6669c6019031e2aa19ad5d3c61e6e375f583c5aa2b0c4c0#5242880 16:42:03.949 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 16:42:03.949 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get blob id' received from client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:03.949 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob a9df54dba0d08e22b6669c6019031e2aa19ad5d3c61e6e375f583c5aa2b0c4c0#5242880 for client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:03.949 DEBUG [primary-1] CommunicationObserver.java:130 Binary with size 5242880 sent to client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:03.949 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob a9df54dba0d08e22b6669c6019031e2aa19ad5d3c61e6e375f583c5aa2b0c4c0#5242880 to client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:03.971 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob a9df54dba0d08e22b6669c6019031e2aa19ad5d3c61e6e375f583c5aa2b0c4c0#5242880 to client 1af42e6d-31ae-4caf-bc83-bac393fd0210 16:42:04.018 DEBUG [standby-run-23] StandbyClient.java:157 Channel closed 16:42:04.018 ERROR [standby-run-23] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to load remote blob a9df54dba0d08e22b6669c6019031e2aa19ad5d3c61e6e375f583c5aa2b0c4c0#5242880 at /root/server/#testBlob in 60ms. Please increase the timeout and try again. at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchAndStoreBlob(StandbyDiff.java:208) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:196) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:181) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:169) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeAdded(StandbyDiff.java:94) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeAdded(CancelableDiff.java:76) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:562) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:566) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:101) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.ExternalPrivateStoreIT.testSyncFailingDueToTooShortTimeout(ExternalPrivateStoreIT.java:86) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 16:42:06.258 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:42:06.258 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:42:06.259 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:42:06.259 DEBUG [main] StandbyServer.java:231 Boss group shut down 16:42:06.261 DEBUG [main] StandbyServer.java:237 Worker group shut down 16:42:06.262 INFO [main] DataStoreTestBase.java:132 Test end: testSyncFailingDueToTooShortTimeout 16:42:06.527 INFO [main] FileStore.java:466 TarMK closed: target\junit5486475792104479439\junit3676281828139618322 16:42:06.564 INFO [main] FileStore.java:466 TarMK closed: target\junit5486475792104479439\junit7610614532259530189 16:42:06.596 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit3188475119481759633\junit4974002286183265593, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:42:06.596 INFO [main] FileStore.java:240 TarMK opened: target\junit3188475119481759633\junit4974002286183265593 (mmap=false) 16:42:06.660 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit3188475119481759633\junit6263869421348268703, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:42:06.663 INFO [main] FileStore.java:240 TarMK opened: target\junit3188475119481759633\junit6263869421348268703 (mmap=false) 16:42:06.665 INFO [main] DataStoreTestBase.java:127 Test begin: testResilientSync 16:42:06.665 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 16:42:06.665 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 16:42:06.781 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x64143835]' 16:42:06.781 DEBUG [main] StandbyServer.java:249 Binding was successful 16:42:06.796 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 7266f33c-cec5-4cb3-9737-1a07fc1862e2 for current head 16:42:06.796 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61271 is allowed 16:42:06.796 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 16:42:06.796 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 7266f33c-cec5-4cb3-9737-1a07fc1862e2 16:42:06.796 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 7266f33c-cec5-4cb3-9737-1a07fc1862e2 16:42:10.800 ERROR [standby-run-24] StandbyClientSyncExecution.java:67 Unable to fetch remote head 16:42:10.800 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 7266f33c-cec5-4cb3-9737-1a07fc1862e2 io.netty.handler.timeout.ReadTimeoutException: null 16:42:10.801 DEBUG [standby-run-24] StandbyClient.java:157 Channel closed 16:42:12.072 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 69b566b1-0d3c-4bbc-a989-d702affdf51a.0000000c to client 7266f33c-cec5-4cb3-9737-1a07fc1862e2 16:42:12.072 WARN [primary-1] ExceptionHandler.java:31 Exception caught on the server java.io.IOException: Eine bestehende Verbindung wurde softwaregesteuert durch den Hostcomputer abgebrochen at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[na:1.8.0_151] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) ~[na:1.8.0_151] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_151] at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_151] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_151] at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288) ~[netty-buffer-4.1.14.Final.jar:4.1.14.Final] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106) ~[netty-buffer-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:372) ~[netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) ~[netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 16:42:13.018 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:42:13.018 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:42:13.018 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:42:13.018 DEBUG [main] StandbyServer.java:231 Boss group shut down 16:42:13.018 DEBUG [main] StandbyServer.java:237 Worker group shut down 16:42:13.065 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x4575259d]' 16:42:13.067 DEBUG [main] StandbyServer.java:249 Binding was successful 16:42:13.070 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client d6c82210-7dae-40f4-98e0-e51069aa4167 for current head 16:42:13.071 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61314 is allowed 16:42:13.071 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 16:42:13.071 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.072 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.072 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 69b566b1-0d3c-4bbc-a989-d702affdf51a.0000000c to client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.072 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 16:42:13.072 DEBUG [standby-run-25] StandbyClientSyncExecution.java:103 Found missing segment 69b566b1-0d3c-4bbc-a989-d702affdf51a 16:42:13.072 DEBUG [standby-run-25] StandbyClientSyncExecution.java:124 Inspecting segment 69b566b1-0d3c-4bbc-a989-d702affdf51a 16:42:13.072 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client d6c82210-7dae-40f4-98e0-e51069aa4167 for references of segment 69b566b1-0d3c-4bbc-a989-d702affdf51a 16:42:13.072 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 16:42:13.072 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 69b566b1-0d3c-4bbc-a989-d702affdf51a for client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.072 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 69b566b1-0d3c-4bbc-a989-d702affdf51a to client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.072 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:42:13.072 DEBUG [standby-run-25] StandbyClientSyncExecution.java:184 Found reference from 69b566b1-0d3c-4bbc-a989-d702affdf51a to c5277900-2213-4956-a40c-403eea989f66 16:42:13.072 DEBUG [standby-run-25] StandbyClientSyncExecution.java:124 Inspecting segment c5277900-2213-4956-a40c-403eea989f66 16:42:13.072 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client d6c82210-7dae-40f4-98e0-e51069aa4167 for references of segment c5277900-2213-4956-a40c-403eea989f66 16:42:13.072 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 16:42:13.072 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment c5277900-2213-4956-a40c-403eea989f66 for client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.072 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment c5277900-2213-4956-a40c-403eea989f66 to client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.072 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:42:13.072 INFO [standby-run-25] StandbyClientSyncExecution.java:196 Copying data segment c5277900-2213-4956-a40c-403eea989f66 from primary 16:42:13.072 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client d6c82210-7dae-40f4-98e0-e51069aa4167 for segment c5277900-2213-4956-a40c-403eea989f66 16:42:13.072 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 16:42:13.072 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.072 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment c5277900-2213-4956-a40c-403eea989f66 for client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.072 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.072 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment c5277900-2213-4956-a40c-403eea989f66 to client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.072 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:42:13.072 INFO [standby-run-25] StandbyClientSyncExecution.java:196 Copying data segment 69b566b1-0d3c-4bbc-a989-d702affdf51a from primary 16:42:13.072 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client d6c82210-7dae-40f4-98e0-e51069aa4167 for segment 69b566b1-0d3c-4bbc-a989-d702affdf51a 16:42:13.072 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 16:42:13.072 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.072 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 69b566b1-0d3c-4bbc-a989-d702affdf51a for client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.072 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.072 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 69b566b1-0d3c-4bbc-a989-d702affdf51a to client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.072 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:42:13.087 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client d6c82210-7dae-40f4-98e0-e51069aa4167 for blob 957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880 16:42:13.087 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 16:42:13.087 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get blob id' received from client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.087 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob 957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880 for client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.087 DEBUG [primary-1] CommunicationObserver.java:130 Binary with size 5242880 sent to client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.087 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob 957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880 to client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.087 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880 to client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.134 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:13.134 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880 16:42:13.134 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880 to client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.134 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880.tmp 16:42:13.172 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:13.172 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880 16:42:13.172 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880 to client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.172 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880.tmp 16:42:13.203 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:13.203 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880 16:42:13.219 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880 to client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.219 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880.tmp 16:42:13.250 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:13.250 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880 16:42:13.250 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880.tmp 16:42:13.250 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880 to client d6c82210-7dae-40f4-98e0-e51069aa4167 16:42:13.303 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:13.303 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880 16:42:13.303 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880.tmp 16:42:13.303 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob 957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880 16:42:13.372 DEBUG [standby-run-25] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\957062972cf47e88c88ced48c1bdda109cd00a537284bb781a364698e8ebe1ab#5242880.tmp 16:42:13.535 INFO [standby-run-25] StandbyClientSyncExecution.java:82 updated head state successfully: true in 463ms. 16:42:13.535 DEBUG [standby-run-25] StandbyClient.java:157 Channel closed 16:42:15.737 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:42:15.737 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:42:15.737 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:42:15.737 DEBUG [main] StandbyServer.java:231 Boss group shut down 16:42:15.737 DEBUG [main] StandbyServer.java:237 Worker group shut down 16:42:15.753 INFO [main] DataStoreTestBase.java:132 Test end: testResilientSync 16:42:15.806 INFO [main] FileStore.java:466 TarMK closed: target\junit3188475119481759633\junit6263869421348268703 16:42:15.837 INFO [main] FileStore.java:466 TarMK closed: target\junit3188475119481759633\junit4974002286183265593 16:42:15.869 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit4808260247270875766\junit6206188993749882264, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:42:15.872 INFO [main] FileStore.java:240 TarMK opened: target\junit4808260247270875766\junit6206188993749882264 (mmap=false) 16:42:15.875 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit4808260247270875766\junit610925834712247328, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:42:15.875 INFO [main] FileStore.java:240 TarMK opened: target\junit4808260247270875766\junit610925834712247328 (mmap=false) 16:42:15.875 INFO [main] DataStoreTestBase.java:127 Test begin: testProxySkippedBytes 16:42:15.875 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 16:42:15.875 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 16:42:16.322 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xe65f6c42]' 16:42:16.322 DEBUG [main] StandbyServer.java:249 Binding was successful 16:42:16.353 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client f2bfc50d-2261-4a89-8bc4-c5d82d6aec8b for current head 16:42:16.375 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61405 is allowed 16:42:16.391 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 16:42:16.391 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client f2bfc50d-2261-4a89-8bc4-c5d82d6aec8b 16:42:16.391 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client f2bfc50d-2261-4a89-8bc4-c5d82d6aec8b 16:42:16.391 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 8ad1fbf5-e053-4541-ac72-e6740b83ab9a.0000000c to client f2bfc50d-2261-4a89-8bc4-c5d82d6aec8b 16:42:16.391 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 16:42:16.391 DEBUG [standby-run-26] StandbyClientSyncExecution.java:103 Found missing segment 8ad1fbf5-e053-4541-ac72-e6740b83ab9a 16:42:16.391 DEBUG [standby-run-26] StandbyClientSyncExecution.java:124 Inspecting segment 8ad1fbf5-e053-4541-ac72-e6740b83ab9a 16:42:16.391 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client f2bfc50d-2261-4a89-8bc4-c5d82d6aec8b for references of segment 8ad1fbf5-e053-4541-ac72-e6740b83ab9a 16:42:16.391 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 16:42:16.391 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 8ad1fbf5-e053-4541-ac72-e6740b83ab9a for client f2bfc50d-2261-4a89-8bc4-c5d82d6aec8b 16:42:16.391 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 8ad1fbf5-e053-4541-ac72-e6740b83ab9a to client f2bfc50d-2261-4a89-8bc4-c5d82d6aec8b 16:42:21.411 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client f2bfc50d-2261-4a89-8bc4-c5d82d6aec8b io.netty.handler.timeout.ReadTimeoutException: null 16:42:21.411 DEBUG [standby-run-26] StandbyClient.java:157 Channel closed 16:42:21.411 ERROR [standby-run-26] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment 8ad1fbf5-e053-4541-ac72-e6740b83ab9a from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:346) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxySkippedBytes(DataStoreTestBase.java:296) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 16:42:23.629 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:42:23.644 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 47b67ace-2e33-43ec-9e9d-90d515e94649 for current head 16:42:23.644 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:61432 is allowed 16:42:23.644 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 16:42:23.644 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.644 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.644 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 8ad1fbf5-e053-4541-ac72-e6740b83ab9a.0000000c to client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.644 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 16:42:23.644 DEBUG [standby-run-27] StandbyClientSyncExecution.java:103 Found missing segment 8ad1fbf5-e053-4541-ac72-e6740b83ab9a 16:42:23.644 DEBUG [standby-run-27] StandbyClientSyncExecution.java:124 Inspecting segment 8ad1fbf5-e053-4541-ac72-e6740b83ab9a 16:42:23.644 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 47b67ace-2e33-43ec-9e9d-90d515e94649 for references of segment 8ad1fbf5-e053-4541-ac72-e6740b83ab9a 16:42:23.644 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 16:42:23.644 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 8ad1fbf5-e053-4541-ac72-e6740b83ab9a for client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.644 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 8ad1fbf5-e053-4541-ac72-e6740b83ab9a to client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.644 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:42:23.644 DEBUG [standby-run-27] StandbyClientSyncExecution.java:184 Found reference from 8ad1fbf5-e053-4541-ac72-e6740b83ab9a to 0cb8a4d9-5475-4f9f-a42c-93c44cdb1b12 16:42:23.660 DEBUG [standby-run-27] StandbyClientSyncExecution.java:124 Inspecting segment 0cb8a4d9-5475-4f9f-a42c-93c44cdb1b12 16:42:23.660 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 47b67ace-2e33-43ec-9e9d-90d515e94649 for references of segment 0cb8a4d9-5475-4f9f-a42c-93c44cdb1b12 16:42:23.660 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 16:42:23.660 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 0cb8a4d9-5475-4f9f-a42c-93c44cdb1b12 for client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.660 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 0cb8a4d9-5475-4f9f-a42c-93c44cdb1b12 to client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.660 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:42:23.660 INFO [standby-run-27] StandbyClientSyncExecution.java:196 Copying data segment 0cb8a4d9-5475-4f9f-a42c-93c44cdb1b12 from primary 16:42:23.660 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 47b67ace-2e33-43ec-9e9d-90d515e94649 for segment 0cb8a4d9-5475-4f9f-a42c-93c44cdb1b12 16:42:23.660 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 16:42:23.660 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.660 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 0cb8a4d9-5475-4f9f-a42c-93c44cdb1b12 for client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.660 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.660 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 0cb8a4d9-5475-4f9f-a42c-93c44cdb1b12 to client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.660 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:42:23.660 INFO [standby-run-27] StandbyClientSyncExecution.java:196 Copying data segment 8ad1fbf5-e053-4541-ac72-e6740b83ab9a from primary 16:42:23.660 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 47b67ace-2e33-43ec-9e9d-90d515e94649 for segment 8ad1fbf5-e053-4541-ac72-e6740b83ab9a 16:42:23.660 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 16:42:23.660 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.660 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 8ad1fbf5-e053-4541-ac72-e6740b83ab9a for client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.660 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.660 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 8ad1fbf5-e053-4541-ac72-e6740b83ab9a to client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.660 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:42:23.660 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 47b67ace-2e33-43ec-9e9d-90d515e94649 for blob c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880 16:42:23.660 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 16:42:23.660 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.660 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880 for client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.660 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 5242880 sent to client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.660 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880 to client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.676 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880 to client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.713 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:23.713 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880 16:42:23.713 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880.tmp 16:42:23.713 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880 to client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.776 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:23.776 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880 16:42:23.779 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880.tmp 16:42:23.782 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880 to client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.813 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:23.813 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880 16:42:23.813 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880.tmp 16:42:23.813 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880 to client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.877 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:23.877 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880 16:42:23.880 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880.tmp 16:42:23.882 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880 to client 47b67ace-2e33-43ec-9e9d-90d515e94649 16:42:23.914 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:23.914 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880 16:42:23.929 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880.tmp 16:42:23.929 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880 16:42:24.005 DEBUG [standby-run-27] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\c55e70bc1a0bf4cbe2aa9a7d23e7f2c6d567904a2b72bfa4515876203d3bbc4d#5242880.tmp 16:42:24.078 INFO [standby-run-27] StandbyClientSyncExecution.java:82 updated head state successfully: true in 434ms. 16:42:24.078 DEBUG [standby-run-27] StandbyClient.java:157 Channel closed 16:42:26.295 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:42:26.295 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:42:26.295 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:42:26.295 DEBUG [main] StandbyServer.java:231 Boss group shut down 16:42:26.295 DEBUG [main] StandbyServer.java:237 Worker group shut down 16:42:26.310 INFO [main] DataStoreTestBase.java:132 Test end: testProxySkippedBytes 16:42:26.341 INFO [main] FileStore.java:466 TarMK closed: target\junit4808260247270875766\junit610925834712247328 16:42:26.395 INFO [main] FileStore.java:466 TarMK closed: target\junit4808260247270875766\junit6206188993749882264 16:42:26.426 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit2992766224674138271\junit1995391762490828046, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:42:26.426 INFO [main] FileStore.java:240 TarMK opened: target\junit2992766224674138271\junit1995391762490828046 (mmap=false) 16:42:26.442 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit2992766224674138271\junit4595631957442791333, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:42:26.442 INFO [main] FileStore.java:240 TarMK opened: target\junit2992766224674138271\junit4595631957442791333 (mmap=false) 16:42:26.442 INFO [main] DataStoreTestBase.java:127 Test begin: testSyncUpdatedBinaryProperty 16:42:26.442 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 16:42:26.442 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 16:42:26.494 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x2902e31e]' 16:42:26.494 DEBUG [main] StandbyServer.java:249 Binding was successful 16:42:27.009 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 65ed551b-6218-4dcc-82e3-51c6caf588a2 for current head 16:42:27.009 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61472 is allowed 16:42:27.009 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 16:42:27.009 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.009 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.009 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head e92f1ae7-0701-4977-aafd-90740dbb412a.0000000c to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.009 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 16:42:27.009 DEBUG [standby-run-28] StandbyClientSyncExecution.java:103 Found missing segment e92f1ae7-0701-4977-aafd-90740dbb412a 16:42:27.009 DEBUG [standby-run-28] StandbyClientSyncExecution.java:124 Inspecting segment e92f1ae7-0701-4977-aafd-90740dbb412a 16:42:27.009 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 65ed551b-6218-4dcc-82e3-51c6caf588a2 for references of segment e92f1ae7-0701-4977-aafd-90740dbb412a 16:42:27.009 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 16:42:27.009 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment e92f1ae7-0701-4977-aafd-90740dbb412a for client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.024 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment e92f1ae7-0701-4977-aafd-90740dbb412a to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.024 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:42:27.024 DEBUG [standby-run-28] StandbyClientSyncExecution.java:184 Found reference from e92f1ae7-0701-4977-aafd-90740dbb412a to ba61ddce-a38e-4b68-ae09-157af14dfe3c 16:42:27.024 DEBUG [standby-run-28] StandbyClientSyncExecution.java:124 Inspecting segment ba61ddce-a38e-4b68-ae09-157af14dfe3c 16:42:27.024 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 65ed551b-6218-4dcc-82e3-51c6caf588a2 for references of segment ba61ddce-a38e-4b68-ae09-157af14dfe3c 16:42:27.024 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 16:42:27.024 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment ba61ddce-a38e-4b68-ae09-157af14dfe3c for client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.024 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment ba61ddce-a38e-4b68-ae09-157af14dfe3c to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.024 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:42:27.024 INFO [standby-run-28] StandbyClientSyncExecution.java:196 Copying data segment ba61ddce-a38e-4b68-ae09-157af14dfe3c from primary 16:42:27.024 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 65ed551b-6218-4dcc-82e3-51c6caf588a2 for segment ba61ddce-a38e-4b68-ae09-157af14dfe3c 16:42:27.024 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 16:42:27.024 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.024 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment ba61ddce-a38e-4b68-ae09-157af14dfe3c for client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.024 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.024 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment ba61ddce-a38e-4b68-ae09-157af14dfe3c to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.024 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:42:27.024 INFO [standby-run-28] StandbyClientSyncExecution.java:196 Copying data segment e92f1ae7-0701-4977-aafd-90740dbb412a from primary 16:42:27.024 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 65ed551b-6218-4dcc-82e3-51c6caf588a2 for segment e92f1ae7-0701-4977-aafd-90740dbb412a 16:42:27.024 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 16:42:27.024 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.024 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment e92f1ae7-0701-4977-aafd-90740dbb412a for client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.024 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.024 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment e92f1ae7-0701-4977-aafd-90740dbb412a to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.024 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:42:27.024 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 65ed551b-6218-4dcc-82e3-51c6caf588a2 for blob 93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880 16:42:27.024 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 16:42:27.024 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get blob id' received from client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.024 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob 93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880 for client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.024 DEBUG [primary-1] CommunicationObserver.java:130 Binary with size 5242880 sent to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.024 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob 93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880 to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.040 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880 to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.089 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:27.090 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880 16:42:27.092 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880 to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.092 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880.tmp 16:42:27.125 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:27.125 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880 16:42:27.140 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880.tmp 16:42:27.140 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880 to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.194 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:27.194 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880 16:42:27.194 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880.tmp 16:42:27.194 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880 to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.256 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:27.256 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880 16:42:27.256 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880.tmp 16:42:27.256 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880 to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.309 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:27.309 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880 16:42:27.309 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880.tmp 16:42:27.325 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob 93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880 16:42:27.394 DEBUG [standby-run-28] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\93dd10800a4bb50e49dd4d3e8f976fd51a688699d8e18a74072c59ccd4af795c#5242880.tmp 16:42:27.494 INFO [standby-run-28] StandbyClientSyncExecution.java:82 updated head state successfully: true in 485ms. 16:42:27.494 DEBUG [standby-run-28] StandbyClient.java:157 Channel closed 16:42:27.694 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client 65ed551b-6218-4dcc-82e3-51c6caf588a2 for current head 16:42:27.709 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:61474 is allowed 16:42:27.709 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 16:42:27.709 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.709 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.709 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 3508ae8b-f9b4-42ac-afc8-0462443d2997.00000006 to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.709 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 16:42:27.709 DEBUG [standby-run-29] StandbyClientSyncExecution.java:103 Found missing segment 3508ae8b-f9b4-42ac-afc8-0462443d2997 16:42:27.709 DEBUG [standby-run-29] StandbyClientSyncExecution.java:124 Inspecting segment 3508ae8b-f9b4-42ac-afc8-0462443d2997 16:42:27.709 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 65ed551b-6218-4dcc-82e3-51c6caf588a2 for references of segment 3508ae8b-f9b4-42ac-afc8-0462443d2997 16:42:27.709 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 16:42:27.709 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 3508ae8b-f9b4-42ac-afc8-0462443d2997 for client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.709 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 3508ae8b-f9b4-42ac-afc8-0462443d2997 to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.709 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 16:42:27.709 INFO [standby-run-29] StandbyClientSyncExecution.java:196 Copying data segment 3508ae8b-f9b4-42ac-afc8-0462443d2997 from primary 16:42:27.709 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 65ed551b-6218-4dcc-82e3-51c6caf588a2 for segment 3508ae8b-f9b4-42ac-afc8-0462443d2997 16:42:27.709 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 16:42:27.709 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.709 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 3508ae8b-f9b4-42ac-afc8-0462443d2997 for client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.709 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 352 sent to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.709 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 3508ae8b-f9b4-42ac-afc8-0462443d2997 to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.709 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 16:42:27.709 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client 65ed551b-6218-4dcc-82e3-51c6caf588a2 for blob 2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880 16:42:27.709 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 16:42:27.725 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.725 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880 for client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.725 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 5242880 sent to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.725 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880 to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.725 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880 to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.772 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:27.772 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880 16:42:27.772 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880.tmp 16:42:27.772 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880 to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.810 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880 to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.810 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:27.810 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880 16:42:27.810 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880.tmp 16:42:27.841 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:27.857 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880 16:42:27.857 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880.tmp 16:42:27.857 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880 to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.926 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:27.926 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880 16:42:27.926 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880.tmp 16:42:27.926 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880 to client 65ed551b-6218-4dcc-82e3-51c6caf588a2 16:42:27.973 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:27.973 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880 16:42:27.973 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880.tmp 16:42:27.973 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob 2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880 16:42:28.042 DEBUG [standby-run-29] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\2ab34f1c1e5dd42bd9dd0c9d8e2ba654a31ad7645da996b8bb61de84b38bea8b#5242880.tmp 16:42:28.126 INFO [standby-run-29] StandbyClientSyncExecution.java:82 updated head state successfully: true in 417ms. 16:42:28.126 DEBUG [standby-run-29] StandbyClient.java:157 Channel closed 16:42:30.431 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:42:30.431 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:42:30.431 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:42:30.431 DEBUG [main] StandbyServer.java:231 Boss group shut down 16:42:30.431 DEBUG [main] StandbyServer.java:237 Worker group shut down 16:42:30.431 INFO [main] DataStoreTestBase.java:132 Test end: testSyncUpdatedBinaryProperty 16:42:30.462 INFO [main] FileStore.java:466 TarMK closed: target\junit2992766224674138271\junit4595631957442791333 16:42:30.500 INFO [main] FileStore.java:466 TarMK closed: target\junit2992766224674138271\junit1995391762490828046 16:42:30.531 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit3790215788891848901\junit8556285755597893098, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:42:30.531 INFO [main] FileStore.java:240 TarMK opened: target\junit3790215788891848901\junit8556285755597893098 (mmap=false) 16:42:30.531 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit3790215788891848901\junit7441377247389716762, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:42:30.531 INFO [main] FileStore.java:240 TarMK opened: target\junit3790215788891848901\junit7441377247389716762 (mmap=false) 16:42:30.547 INFO [main] DataStoreTestBase.java:127 Test begin: testSync 16:42:30.547 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 16:42:30.547 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 16:42:30.647 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x6970210a]' 16:42:30.647 DEBUG [main] StandbyServer.java:249 Binding was successful 16:42:30.900 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client ee478873-5947-4116-963f-67b601c55a5d for current head 16:42:30.900 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61514 is allowed 16:42:30.900 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 16:42:30.900 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.900 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.900 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head e4026860-c840-473b-a2d0-fb92bb3aace2.0000000c to client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.900 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 16:42:30.900 DEBUG [standby-run-30] StandbyClientSyncExecution.java:103 Found missing segment e4026860-c840-473b-a2d0-fb92bb3aace2 16:42:30.900 DEBUG [standby-run-30] StandbyClientSyncExecution.java:124 Inspecting segment e4026860-c840-473b-a2d0-fb92bb3aace2 16:42:30.900 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client ee478873-5947-4116-963f-67b601c55a5d for references of segment e4026860-c840-473b-a2d0-fb92bb3aace2 16:42:30.900 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 16:42:30.900 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment e4026860-c840-473b-a2d0-fb92bb3aace2 for client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.900 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment e4026860-c840-473b-a2d0-fb92bb3aace2 to client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.900 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:42:30.900 DEBUG [standby-run-30] StandbyClientSyncExecution.java:184 Found reference from e4026860-c840-473b-a2d0-fb92bb3aace2 to fb8127e5-e389-462b-ad0b-0b5b69b229b9 16:42:30.900 DEBUG [standby-run-30] StandbyClientSyncExecution.java:124 Inspecting segment fb8127e5-e389-462b-ad0b-0b5b69b229b9 16:42:30.900 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client ee478873-5947-4116-963f-67b601c55a5d for references of segment fb8127e5-e389-462b-ad0b-0b5b69b229b9 16:42:30.900 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 16:42:30.900 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment fb8127e5-e389-462b-ad0b-0b5b69b229b9 for client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.900 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment fb8127e5-e389-462b-ad0b-0b5b69b229b9 to client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.916 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:42:30.916 INFO [standby-run-30] StandbyClientSyncExecution.java:196 Copying data segment fb8127e5-e389-462b-ad0b-0b5b69b229b9 from primary 16:42:30.916 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client ee478873-5947-4116-963f-67b601c55a5d for segment fb8127e5-e389-462b-ad0b-0b5b69b229b9 16:42:30.916 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 16:42:30.916 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.916 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment fb8127e5-e389-462b-ad0b-0b5b69b229b9 for client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.916 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.916 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment fb8127e5-e389-462b-ad0b-0b5b69b229b9 to client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.916 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:42:30.916 INFO [standby-run-30] StandbyClientSyncExecution.java:196 Copying data segment e4026860-c840-473b-a2d0-fb92bb3aace2 from primary 16:42:30.916 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client ee478873-5947-4116-963f-67b601c55a5d for segment e4026860-c840-473b-a2d0-fb92bb3aace2 16:42:30.916 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 16:42:30.916 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.916 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment e4026860-c840-473b-a2d0-fb92bb3aace2 for client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.916 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.916 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment e4026860-c840-473b-a2d0-fb92bb3aace2 to client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.916 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:42:30.916 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client ee478873-5947-4116-963f-67b601c55a5d for blob 29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880 16:42:30.916 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 16:42:30.916 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get blob id' received from client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.916 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob 29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880 for client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.916 DEBUG [primary-1] CommunicationObserver.java:130 Binary with size 5242880 sent to client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.916 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob 29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880 to client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.916 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880 to client ee478873-5947-4116-963f-67b601c55a5d 16:42:30.963 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:30.963 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880 16:42:30.963 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880.tmp 16:42:30.963 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880 to client ee478873-5947-4116-963f-67b601c55a5d 16:42:31.000 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:31.000 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880 16:42:31.000 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880.tmp 16:42:31.000 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880 to client ee478873-5947-4116-963f-67b601c55a5d 16:42:31.031 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:31.031 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880 16:42:31.031 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880.tmp 16:42:31.031 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880 to client ee478873-5947-4116-963f-67b601c55a5d 16:42:31.062 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:31.062 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880 16:42:31.062 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880 to client ee478873-5947-4116-963f-67b601c55a5d 16:42:31.062 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880.tmp 16:42:31.100 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:31.100 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880 16:42:31.100 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880.tmp 16:42:31.100 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob 29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880 16:42:31.163 DEBUG [standby-run-30] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\29c21dc6712fe9b325768f93305e38fa0598ee8f82840ff11c1edea03aa5ed65#5242880.tmp 16:42:31.248 INFO [standby-run-30] StandbyClientSyncExecution.java:82 updated head state successfully: true in 348ms. 16:42:31.248 DEBUG [standby-run-30] StandbyClient.java:157 Channel closed 16:42:33.464 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:42:33.464 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:42:33.464 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:42:33.464 DEBUG [main] StandbyServer.java:231 Boss group shut down 16:42:33.464 DEBUG [main] StandbyServer.java:237 Worker group shut down 16:42:33.464 INFO [main] DataStoreTestBase.java:132 Test end: testSync 16:42:33.502 INFO [main] FileStore.java:466 TarMK closed: target\junit3790215788891848901\junit7441377247389716762 16:42:33.533 INFO [main] FileStore.java:466 TarMK closed: target\junit3790215788891848901\junit8556285755597893098 16:42:33.564 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit3407839054719107290\junit5502989264278706793, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:42:33.564 INFO [main] FileStore.java:240 TarMK opened: target\junit3407839054719107290\junit5502989264278706793 (mmap=false) 16:42:33.564 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit3407839054719107290\junit116810712867618140, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:42:33.580 INFO [main] FileStore.java:240 TarMK opened: target\junit3407839054719107290\junit116810712867618140 (mmap=false) 16:42:33.580 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedIntermediateByteChange2 16:42:33.580 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 16:42:33.580 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 16:42:33.780 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xf9037bc2]' 16:42:33.796 DEBUG [main] StandbyServer.java:249 Binding was successful 16:42:33.834 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 7b3117ff-dd9e-43fc-94df-7987413e3a50 for current head 16:42:33.849 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61602 is allowed 16:42:33.849 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 16:42:33.849 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.849 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.849 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head d11f3ba3-b15d-4b25-ae26-bfb1276292d4.0000000c to client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.849 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 16:42:33.849 DEBUG [standby-run-31] StandbyClientSyncExecution.java:103 Found missing segment d11f3ba3-b15d-4b25-ae26-bfb1276292d4 16:42:33.849 DEBUG [standby-run-31] StandbyClientSyncExecution.java:124 Inspecting segment d11f3ba3-b15d-4b25-ae26-bfb1276292d4 16:42:33.849 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 7b3117ff-dd9e-43fc-94df-7987413e3a50 for references of segment d11f3ba3-b15d-4b25-ae26-bfb1276292d4 16:42:33.849 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 16:42:33.849 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment d11f3ba3-b15d-4b25-ae26-bfb1276292d4 for client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.849 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment d11f3ba3-b15d-4b25-ae26-bfb1276292d4 to client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.849 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:42:33.849 DEBUG [standby-run-31] StandbyClientSyncExecution.java:184 Found reference from d11f3ba3-b15d-4b25-ae26-bfb1276292d4 to 78c71501-f732-48b8-a2dc-94b0bcb02563 16:42:33.849 DEBUG [standby-run-31] StandbyClientSyncExecution.java:124 Inspecting segment 78c71501-f732-48b8-a2dc-94b0bcb02563 16:42:33.849 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 7b3117ff-dd9e-43fc-94df-7987413e3a50 for references of segment 78c71501-f732-48b8-a2dc-94b0bcb02563 16:42:33.849 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 16:42:33.849 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 78c71501-f732-48b8-a2dc-94b0bcb02563 for client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.849 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 78c71501-f732-48b8-a2dc-94b0bcb02563 to client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.849 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:42:33.849 INFO [standby-run-31] StandbyClientSyncExecution.java:196 Copying data segment 78c71501-f732-48b8-a2dc-94b0bcb02563 from primary 16:42:33.849 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 7b3117ff-dd9e-43fc-94df-7987413e3a50 for segment 78c71501-f732-48b8-a2dc-94b0bcb02563 16:42:33.849 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 16:42:33.849 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.849 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 78c71501-f732-48b8-a2dc-94b0bcb02563 for client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.849 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.849 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 78c71501-f732-48b8-a2dc-94b0bcb02563 to client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.849 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:42:33.849 INFO [standby-run-31] StandbyClientSyncExecution.java:196 Copying data segment d11f3ba3-b15d-4b25-ae26-bfb1276292d4 from primary 16:42:33.849 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 7b3117ff-dd9e-43fc-94df-7987413e3a50 for segment d11f3ba3-b15d-4b25-ae26-bfb1276292d4 16:42:33.865 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 16:42:33.865 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.865 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment d11f3ba3-b15d-4b25-ae26-bfb1276292d4 for client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.865 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.865 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment d11f3ba3-b15d-4b25-ae26-bfb1276292d4 to client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.865 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:42:33.865 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 7b3117ff-dd9e-43fc-94df-7987413e3a50 for blob e3e179fd67b7052055f069a8b77a2ae072a6bcb5a56c503e444073698f1ce5dc#5242880 16:42:33.865 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 16:42:33.865 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get blob id' received from client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.865 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob e3e179fd67b7052055f069a8b77a2ae072a6bcb5a56c503e444073698f1ce5dc#5242880 for client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.865 DEBUG [primary-1] CommunicationObserver.java:130 Binary with size 5242880 sent to client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.865 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob e3e179fd67b7052055f069a8b77a2ae072a6bcb5a56c503e444073698f1ce5dc#5242880 to client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.865 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob e3e179fd67b7052055f069a8b77a2ae072a6bcb5a56c503e444073698f1ce5dc#5242880 to client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:33.881 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 7b3117ff-dd9e-43fc-94df-7987413e3a50 io.netty.handler.codec.compression.DecompressionException: mismatching checksum: a282b566 (expected: 71553bd3) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 16:42:33.903 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob e3e179fd67b7052055f069a8b77a2ae072a6bcb5a56c503e444073698f1ce5dc#5242880 to client 7b3117ff-dd9e-43fc-94df-7987413e3a50 16:42:38.867 DEBUG [standby-run-31] StandbyClient.java:157 Channel closed 16:42:38.867 ERROR [standby-run-31] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to load remote blob e3e179fd67b7052055f069a8b77a2ae072a6bcb5a56c503e444073698f1ce5dc#5242880 at /root/server/#testBlob in 5000ms. Please increase the timeout and try again. at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchAndStoreBlob(StandbyDiff.java:208) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:196) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:181) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:169) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeAdded(StandbyDiff.java:94) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeAdded(CancelableDiff.java:76) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:562) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:566) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:101) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:346) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByteChange2(DataStoreTestBase.java:326) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 16:42:41.084 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:42:41.352 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 6d06359e-28ed-462b-916f-2364b47c2ea7 for current head 16:42:41.352 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:61630 is allowed 16:42:41.352 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 16:42:41.352 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 6d06359e-28ed-462b-916f-2364b47c2ea7 16:42:41.352 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 6d06359e-28ed-462b-916f-2364b47c2ea7 16:42:41.352 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 57a70072-7039-40a6-a2df-92fde759ac67.00000006 to client 6d06359e-28ed-462b-916f-2364b47c2ea7 16:42:41.352 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 16:42:41.352 DEBUG [standby-run-32] StandbyClientSyncExecution.java:103 Found missing segment 57a70072-7039-40a6-a2df-92fde759ac67 16:42:41.352 DEBUG [standby-run-32] StandbyClientSyncExecution.java:124 Inspecting segment 57a70072-7039-40a6-a2df-92fde759ac67 16:42:41.352 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 6d06359e-28ed-462b-916f-2364b47c2ea7 for references of segment 57a70072-7039-40a6-a2df-92fde759ac67 16:42:41.352 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 16:42:41.352 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 57a70072-7039-40a6-a2df-92fde759ac67 for client 6d06359e-28ed-462b-916f-2364b47c2ea7 16:42:41.352 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 57a70072-7039-40a6-a2df-92fde759ac67 to client 6d06359e-28ed-462b-916f-2364b47c2ea7 16:42:41.368 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:42:41.368 INFO [standby-run-32] StandbyClientSyncExecution.java:196 Copying data segment 57a70072-7039-40a6-a2df-92fde759ac67 from primary 16:42:41.368 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 6d06359e-28ed-462b-916f-2364b47c2ea7 for segment 57a70072-7039-40a6-a2df-92fde759ac67 16:42:41.368 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 16:42:41.368 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 6d06359e-28ed-462b-916f-2364b47c2ea7 16:42:41.368 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 57a70072-7039-40a6-a2df-92fde759ac67 for client 6d06359e-28ed-462b-916f-2364b47c2ea7 16:42:41.368 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 352 sent to client 6d06359e-28ed-462b-916f-2364b47c2ea7 16:42:41.368 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 57a70072-7039-40a6-a2df-92fde759ac67 to client 6d06359e-28ed-462b-916f-2364b47c2ea7 16:42:41.368 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:42:41.368 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 6d06359e-28ed-462b-916f-2364b47c2ea7 for blob cb2b69b38244ea9e83124291240496991139699716cca2bafb845c7420f85f8b#2097152 16:42:41.368 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 16:42:41.368 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client 6d06359e-28ed-462b-916f-2364b47c2ea7 16:42:41.368 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob cb2b69b38244ea9e83124291240496991139699716cca2bafb845c7420f85f8b#2097152 for client 6d06359e-28ed-462b-916f-2364b47c2ea7 16:42:41.368 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 2097152 sent to client 6d06359e-28ed-462b-916f-2364b47c2ea7 16:42:41.368 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob cb2b69b38244ea9e83124291240496991139699716cca2bafb845c7420f85f8b#2097152 to client 6d06359e-28ed-462b-916f-2364b47c2ea7 16:42:41.384 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/2 of size 1048576 from blob cb2b69b38244ea9e83124291240496991139699716cca2bafb845c7420f85f8b#2097152 to client 6d06359e-28ed-462b-916f-2364b47c2ea7 16:42:41.421 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:41.421 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/2 of size 1048576 from blob cb2b69b38244ea9e83124291240496991139699716cca2bafb845c7420f85f8b#2097152 16:42:41.421 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/2 of size 1048576 from blob cb2b69b38244ea9e83124291240496991139699716cca2bafb845c7420f85f8b#2097152 to client 6d06359e-28ed-462b-916f-2364b47c2ea7 16:42:41.421 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\cb2b69b38244ea9e83124291240496991139699716cca2bafb845c7420f85f8b#2097152.tmp 16:42:41.452 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:41.452 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/2 of size 1048576 from blob cb2b69b38244ea9e83124291240496991139699716cca2bafb845c7420f85f8b#2097152 16:42:41.468 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\cb2b69b38244ea9e83124291240496991139699716cca2bafb845c7420f85f8b#2097152.tmp 16:42:41.483 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob cb2b69b38244ea9e83124291240496991139699716cca2bafb845c7420f85f8b#2097152 16:42:41.506 DEBUG [standby-run-32] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\cb2b69b38244ea9e83124291240496991139699716cca2bafb845c7420f85f8b#2097152.tmp 16:42:41.684 INFO [standby-run-32] StandbyClientSyncExecution.java:82 updated head state successfully: true in 332ms. 16:42:41.684 DEBUG [standby-run-32] StandbyClient.java:157 Channel closed 16:42:43.906 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:42:43.906 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:42:43.906 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:42:43.906 DEBUG [main] StandbyServer.java:231 Boss group shut down 16:42:43.906 DEBUG [main] StandbyServer.java:237 Worker group shut down 16:42:43.922 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedIntermediateByteChange2 16:42:43.953 INFO [main] FileStore.java:466 TarMK closed: target\junit3407839054719107290\junit116810712867618140 16:42:43.985 INFO [main] FileStore.java:466 TarMK closed: target\junit3407839054719107290\junit5502989264278706793 16:42:44.007 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit5050867104802020332\junit6916418292871043581, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:42:44.007 INFO [main] FileStore.java:240 TarMK opened: target\junit5050867104802020332\junit6916418292871043581 (mmap=false) 16:42:44.022 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit5050867104802020332\junit7882671870404867891, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:42:44.022 INFO [main] FileStore.java:240 TarMK opened: target\junit5050867104802020332\junit7882671870404867891 (mmap=false) 16:42:44.022 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedIntermediateByteChange 16:42:44.022 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 16:42:44.022 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 16:42:44.206 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x48908723]' 16:42:44.206 DEBUG [main] StandbyServer.java:249 Binding was successful 16:42:44.238 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 12f2832d-a7b7-4518-bdcb-4968b00907e4 for current head 16:42:44.253 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61719 is allowed 16:42:44.253 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 16:42:44.253 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 12f2832d-a7b7-4518-bdcb-4968b00907e4 16:42:44.253 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 12f2832d-a7b7-4518-bdcb-4968b00907e4 16:42:44.253 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 7701ef11-eaa7-4d48-a5b1-5ff1e8d0311a.0000000c to client 12f2832d-a7b7-4518-bdcb-4968b00907e4 16:42:44.253 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 16:42:44.253 DEBUG [standby-run-33] StandbyClientSyncExecution.java:103 Found missing segment 7701ef11-eaa7-4d48-a5b1-5ff1e8d0311a 16:42:44.253 DEBUG [standby-run-33] StandbyClientSyncExecution.java:124 Inspecting segment 7701ef11-eaa7-4d48-a5b1-5ff1e8d0311a 16:42:44.253 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 12f2832d-a7b7-4518-bdcb-4968b00907e4 for references of segment 7701ef11-eaa7-4d48-a5b1-5ff1e8d0311a 16:42:44.253 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 16:42:44.253 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 7701ef11-eaa7-4d48-a5b1-5ff1e8d0311a for client 12f2832d-a7b7-4518-bdcb-4968b00907e4 16:42:44.269 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 7701ef11-eaa7-4d48-a5b1-5ff1e8d0311a to client 12f2832d-a7b7-4518-bdcb-4968b00907e4 16:42:44.269 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 12f2832d-a7b7-4518-bdcb-4968b00907e4 io.netty.handler.codec.compression.DecompressionException: mismatching checksum: a2824ef5 (expected: d03f362e) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 16:42:49.274 DEBUG [standby-run-33] StandbyClient.java:157 Channel closed 16:42:49.274 ERROR [standby-run-33] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment 7701ef11-eaa7-4d48-a5b1-5ff1e8d0311a from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:346) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByteChange(DataStoreTestBase.java:321) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 16:42:51.491 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:42:51.645 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client b1f71563-c0d8-4343-913d-7f5f97a3b612 for current head 16:42:51.645 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:61743 is allowed 16:42:51.645 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 16:42:51.645 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.645 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.645 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 043dc214-623e-415f-ac72-8ccc127e3241.00000006 to client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.645 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 16:42:51.645 DEBUG [standby-run-34] StandbyClientSyncExecution.java:103 Found missing segment 043dc214-623e-415f-ac72-8ccc127e3241 16:42:51.645 DEBUG [standby-run-34] StandbyClientSyncExecution.java:124 Inspecting segment 043dc214-623e-415f-ac72-8ccc127e3241 16:42:51.645 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client b1f71563-c0d8-4343-913d-7f5f97a3b612 for references of segment 043dc214-623e-415f-ac72-8ccc127e3241 16:42:51.645 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 16:42:51.645 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 043dc214-623e-415f-ac72-8ccc127e3241 for client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.645 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 043dc214-623e-415f-ac72-8ccc127e3241 to client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.645 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:42:51.645 DEBUG [standby-run-34] StandbyClientSyncExecution.java:184 Found reference from 043dc214-623e-415f-ac72-8ccc127e3241 to 7701ef11-eaa7-4d48-a5b1-5ff1e8d0311a 16:42:51.645 DEBUG [standby-run-34] StandbyClientSyncExecution.java:184 Found reference from 043dc214-623e-415f-ac72-8ccc127e3241 to 33b5ae7f-a224-463e-a06d-0e2b410ba71a 16:42:51.645 DEBUG [standby-run-34] StandbyClientSyncExecution.java:124 Inspecting segment 7701ef11-eaa7-4d48-a5b1-5ff1e8d0311a 16:42:51.645 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client b1f71563-c0d8-4343-913d-7f5f97a3b612 for references of segment 7701ef11-eaa7-4d48-a5b1-5ff1e8d0311a 16:42:51.645 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 16:42:51.645 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 7701ef11-eaa7-4d48-a5b1-5ff1e8d0311a for client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.645 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 7701ef11-eaa7-4d48-a5b1-5ff1e8d0311a to client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.645 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:42:51.645 DEBUG [standby-run-34] StandbyClientSyncExecution.java:124 Inspecting segment 33b5ae7f-a224-463e-a06d-0e2b410ba71a 16:42:51.645 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client b1f71563-c0d8-4343-913d-7f5f97a3b612 for references of segment 33b5ae7f-a224-463e-a06d-0e2b410ba71a 16:42:51.645 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 16:42:51.645 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 33b5ae7f-a224-463e-a06d-0e2b410ba71a for client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.645 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 33b5ae7f-a224-463e-a06d-0e2b410ba71a to client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.645 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:42:51.645 INFO [standby-run-34] StandbyClientSyncExecution.java:196 Copying data segment 33b5ae7f-a224-463e-a06d-0e2b410ba71a from primary 16:42:51.645 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client b1f71563-c0d8-4343-913d-7f5f97a3b612 for segment 33b5ae7f-a224-463e-a06d-0e2b410ba71a 16:42:51.645 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 16:42:51.645 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.645 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 33b5ae7f-a224-463e-a06d-0e2b410ba71a for client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.645 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.645 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 33b5ae7f-a224-463e-a06d-0e2b410ba71a to client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.645 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:42:51.645 INFO [standby-run-34] StandbyClientSyncExecution.java:196 Copying data segment 7701ef11-eaa7-4d48-a5b1-5ff1e8d0311a from primary 16:42:51.645 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client b1f71563-c0d8-4343-913d-7f5f97a3b612 for segment 7701ef11-eaa7-4d48-a5b1-5ff1e8d0311a 16:42:51.645 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 16:42:51.645 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.645 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 7701ef11-eaa7-4d48-a5b1-5ff1e8d0311a for client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.645 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.645 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 7701ef11-eaa7-4d48-a5b1-5ff1e8d0311a to client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.645 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:42:51.645 INFO [standby-run-34] StandbyClientSyncExecution.java:196 Copying data segment 043dc214-623e-415f-ac72-8ccc127e3241 from primary 16:42:51.645 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client b1f71563-c0d8-4343-913d-7f5f97a3b612 for segment 043dc214-623e-415f-ac72-8ccc127e3241 16:42:51.645 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 16:42:51.645 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.660 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 043dc214-623e-415f-ac72-8ccc127e3241 for client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.660 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 352 sent to client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.660 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 043dc214-623e-415f-ac72-8ccc127e3241 to client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.660 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:42:51.660 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client b1f71563-c0d8-4343-913d-7f5f97a3b612 for blob dab1fb6d231a9cc62605ad84d96d67218c71c8eec77adaefb66ce002492b6b3e#2097152 16:42:51.660 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 16:42:51.660 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.660 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob dab1fb6d231a9cc62605ad84d96d67218c71c8eec77adaefb66ce002492b6b3e#2097152 for client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.660 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 2097152 sent to client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.660 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob dab1fb6d231a9cc62605ad84d96d67218c71c8eec77adaefb66ce002492b6b3e#2097152 to client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.660 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/2 of size 1048576 from blob dab1fb6d231a9cc62605ad84d96d67218c71c8eec77adaefb66ce002492b6b3e#2097152 to client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.709 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:51.709 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/2 of size 1048576 from blob dab1fb6d231a9cc62605ad84d96d67218c71c8eec77adaefb66ce002492b6b3e#2097152 16:42:51.712 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/2 of size 1048576 from blob dab1fb6d231a9cc62605ad84d96d67218c71c8eec77adaefb66ce002492b6b3e#2097152 to client b1f71563-c0d8-4343-913d-7f5f97a3b612 16:42:51.712 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\dab1fb6d231a9cc62605ad84d96d67218c71c8eec77adaefb66ce002492b6b3e#2097152.tmp 16:42:51.745 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:42:51.745 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/2 of size 1048576 from blob dab1fb6d231a9cc62605ad84d96d67218c71c8eec77adaefb66ce002492b6b3e#2097152 16:42:51.745 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\dab1fb6d231a9cc62605ad84d96d67218c71c8eec77adaefb66ce002492b6b3e#2097152.tmp 16:42:51.745 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob dab1fb6d231a9cc62605ad84d96d67218c71c8eec77adaefb66ce002492b6b3e#2097152 16:42:51.792 DEBUG [standby-run-34] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\dab1fb6d231a9cc62605ad84d96d67218c71c8eec77adaefb66ce002492b6b3e#2097152.tmp 16:42:51.886 INFO [standby-run-34] StandbyClientSyncExecution.java:82 updated head state successfully: true in 241ms. 16:42:51.888 DEBUG [standby-run-34] StandbyClient.java:157 Channel closed 16:42:54.131 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:42:54.131 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:42:54.131 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:42:54.131 DEBUG [main] StandbyServer.java:231 Boss group shut down 16:42:54.131 DEBUG [main] StandbyServer.java:237 Worker group shut down 16:42:54.131 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedIntermediateByteChange 16:42:54.178 INFO [main] FileStore.java:466 TarMK closed: target\junit5050867104802020332\junit7882671870404867891 16:42:54.215 INFO [main] FileStore.java:466 TarMK closed: target\junit5050867104802020332\junit6916418292871043581 16:42:54.231 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit2884591739940596404\junit7181127514926271123, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:42:54.231 INFO [main] FileStore.java:240 TarMK opened: target\junit2884591739940596404\junit7181127514926271123 (mmap=false) 16:42:54.231 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit2884591739940596404\junit1580488933204767281, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:42:54.247 INFO [main] FileStore.java:240 TarMK opened: target\junit2884591739940596404\junit1580488933204767281 (mmap=false) 16:42:54.247 INFO [main] DataStoreTestBase.java:127 Test begin: testProxySkippedBytesIntermediateChange 16:42:54.247 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 16:42:54.247 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 16:42:54.416 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x56241c54]' 16:42:54.416 DEBUG [main] StandbyServer.java:249 Binding was successful 16:42:54.463 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client b8bd584d-8e96-4498-a2aa-84e011e70afe for current head 16:42:54.463 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61832 is allowed 16:42:54.463 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 16:42:54.463 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client b8bd584d-8e96-4498-a2aa-84e011e70afe 16:42:54.463 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client b8bd584d-8e96-4498-a2aa-84e011e70afe 16:42:54.463 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head d975ab23-da90-4b85-a844-865411c72caa.0000000c to client b8bd584d-8e96-4498-a2aa-84e011e70afe 16:42:54.463 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 16:42:54.479 DEBUG [standby-run-35] StandbyClientSyncExecution.java:103 Found missing segment d975ab23-da90-4b85-a844-865411c72caa 16:42:54.479 DEBUG [standby-run-35] StandbyClientSyncExecution.java:124 Inspecting segment d975ab23-da90-4b85-a844-865411c72caa 16:42:54.479 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client b8bd584d-8e96-4498-a2aa-84e011e70afe for references of segment d975ab23-da90-4b85-a844-865411c72caa 16:42:54.479 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 16:42:54.479 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment d975ab23-da90-4b85-a844-865411c72caa for client b8bd584d-8e96-4498-a2aa-84e011e70afe 16:42:54.479 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment d975ab23-da90-4b85-a844-865411c72caa to client b8bd584d-8e96-4498-a2aa-84e011e70afe 16:42:59.481 DEBUG [standby-run-35] StandbyClient.java:157 Channel closed 16:42:59.481 ERROR [standby-run-35] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment d975ab23-da90-4b85-a844-865411c72caa from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:346) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxySkippedBytesIntermediateChange(DataStoreTestBase.java:301) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 16:43:01.813 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:43:02.118 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 5733cb8f-8187-426f-afb9-5603ea00cdc3 for current head 16:43:02.118 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:61856 is allowed 16:43:02.118 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 16:43:02.118 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.118 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.118 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head ba1d8ea3-e27c-4188-a068-11f7d6074cb0.00000006 to client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.118 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 16:43:02.118 DEBUG [standby-run-36] StandbyClientSyncExecution.java:103 Found missing segment ba1d8ea3-e27c-4188-a068-11f7d6074cb0 16:43:02.118 DEBUG [standby-run-36] StandbyClientSyncExecution.java:124 Inspecting segment ba1d8ea3-e27c-4188-a068-11f7d6074cb0 16:43:02.118 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 5733cb8f-8187-426f-afb9-5603ea00cdc3 for references of segment ba1d8ea3-e27c-4188-a068-11f7d6074cb0 16:43:02.118 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 16:43:02.118 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment ba1d8ea3-e27c-4188-a068-11f7d6074cb0 for client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.118 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment ba1d8ea3-e27c-4188-a068-11f7d6074cb0 to client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.118 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:43:02.134 DEBUG [standby-run-36] StandbyClientSyncExecution.java:184 Found reference from ba1d8ea3-e27c-4188-a068-11f7d6074cb0 to d975ab23-da90-4b85-a844-865411c72caa 16:43:02.134 DEBUG [standby-run-36] StandbyClientSyncExecution.java:184 Found reference from ba1d8ea3-e27c-4188-a068-11f7d6074cb0 to 0e01fa79-b0e3-4570-a6df-3a052f96783c 16:43:02.134 DEBUG [standby-run-36] StandbyClientSyncExecution.java:124 Inspecting segment d975ab23-da90-4b85-a844-865411c72caa 16:43:02.134 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 5733cb8f-8187-426f-afb9-5603ea00cdc3 for references of segment d975ab23-da90-4b85-a844-865411c72caa 16:43:02.134 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 16:43:02.134 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment d975ab23-da90-4b85-a844-865411c72caa for client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.134 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment d975ab23-da90-4b85-a844-865411c72caa to client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.134 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:43:02.134 DEBUG [standby-run-36] StandbyClientSyncExecution.java:124 Inspecting segment 0e01fa79-b0e3-4570-a6df-3a052f96783c 16:43:02.134 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 5733cb8f-8187-426f-afb9-5603ea00cdc3 for references of segment 0e01fa79-b0e3-4570-a6df-3a052f96783c 16:43:02.134 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 16:43:02.134 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 0e01fa79-b0e3-4570-a6df-3a052f96783c for client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.134 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 0e01fa79-b0e3-4570-a6df-3a052f96783c to client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.134 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:43:02.134 INFO [standby-run-36] StandbyClientSyncExecution.java:196 Copying data segment 0e01fa79-b0e3-4570-a6df-3a052f96783c from primary 16:43:02.134 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 5733cb8f-8187-426f-afb9-5603ea00cdc3 for segment 0e01fa79-b0e3-4570-a6df-3a052f96783c 16:43:02.134 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 16:43:02.134 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.134 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 0e01fa79-b0e3-4570-a6df-3a052f96783c for client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.134 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.134 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 0e01fa79-b0e3-4570-a6df-3a052f96783c to client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.134 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:43:02.134 INFO [standby-run-36] StandbyClientSyncExecution.java:196 Copying data segment d975ab23-da90-4b85-a844-865411c72caa from primary 16:43:02.134 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 5733cb8f-8187-426f-afb9-5603ea00cdc3 for segment d975ab23-da90-4b85-a844-865411c72caa 16:43:02.134 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 16:43:02.134 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.134 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment d975ab23-da90-4b85-a844-865411c72caa for client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.134 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.134 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment d975ab23-da90-4b85-a844-865411c72caa to client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.149 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:43:02.149 INFO [standby-run-36] StandbyClientSyncExecution.java:196 Copying data segment ba1d8ea3-e27c-4188-a068-11f7d6074cb0 from primary 16:43:02.149 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 5733cb8f-8187-426f-afb9-5603ea00cdc3 for segment ba1d8ea3-e27c-4188-a068-11f7d6074cb0 16:43:02.149 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 16:43:02.149 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.149 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment ba1d8ea3-e27c-4188-a068-11f7d6074cb0 for client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.149 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 352 sent to client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.149 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment ba1d8ea3-e27c-4188-a068-11f7d6074cb0 to client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.149 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:43:02.149 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 5733cb8f-8187-426f-afb9-5603ea00cdc3 for blob a86a84321a561de8b7c0e8fcd27915b5f19e99c50d0039a8a38d4767026b1d73#2097152 16:43:02.149 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 16:43:02.149 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.149 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob a86a84321a561de8b7c0e8fcd27915b5f19e99c50d0039a8a38d4767026b1d73#2097152 for client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.149 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 2097152 sent to client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.149 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob a86a84321a561de8b7c0e8fcd27915b5f19e99c50d0039a8a38d4767026b1d73#2097152 to client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.149 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/2 of size 1048576 from blob a86a84321a561de8b7c0e8fcd27915b5f19e99c50d0039a8a38d4767026b1d73#2097152 to client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.181 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:43:02.181 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/2 of size 1048576 from blob a86a84321a561de8b7c0e8fcd27915b5f19e99c50d0039a8a38d4767026b1d73#2097152 16:43:02.181 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/2 of size 1048576 from blob a86a84321a561de8b7c0e8fcd27915b5f19e99c50d0039a8a38d4767026b1d73#2097152 to client 5733cb8f-8187-426f-afb9-5603ea00cdc3 16:43:02.181 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\a86a84321a561de8b7c0e8fcd27915b5f19e99c50d0039a8a38d4767026b1d73#2097152.tmp 16:43:02.218 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:43:02.218 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/2 of size 1048576 from blob a86a84321a561de8b7c0e8fcd27915b5f19e99c50d0039a8a38d4767026b1d73#2097152 16:43:02.218 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\a86a84321a561de8b7c0e8fcd27915b5f19e99c50d0039a8a38d4767026b1d73#2097152.tmp 16:43:02.234 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob a86a84321a561de8b7c0e8fcd27915b5f19e99c50d0039a8a38d4767026b1d73#2097152 16:43:02.265 DEBUG [standby-run-36] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\a86a84321a561de8b7c0e8fcd27915b5f19e99c50d0039a8a38d4767026b1d73#2097152.tmp 16:43:02.433 INFO [standby-run-36] StandbyClientSyncExecution.java:82 updated head state successfully: true in 315ms. 16:43:02.433 DEBUG [standby-run-36] StandbyClient.java:157 Channel closed 16:43:04.636 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:43:04.636 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:43:04.636 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:43:04.636 DEBUG [main] StandbyServer.java:231 Boss group shut down 16:43:04.636 DEBUG [main] StandbyServer.java:237 Worker group shut down 16:43:04.652 INFO [main] DataStoreTestBase.java:132 Test end: testProxySkippedBytesIntermediateChange 16:43:04.699 INFO [main] FileStore.java:466 TarMK closed: target\junit2884591739940596404\junit1580488933204767281 16:43:04.736 INFO [main] FileStore.java:466 TarMK closed: target\junit2884591739940596404\junit7181127514926271123 16:43:04.752 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit6743701212962828640\junit4303984672364308294, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:43:04.768 INFO [main] FileStore.java:240 TarMK opened: target\junit6743701212962828640\junit4303984672364308294 (mmap=false) 16:43:04.768 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit6743701212962828640\junit1349019563279628173, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:43:04.768 INFO [main] FileStore.java:240 TarMK opened: target\junit6743701212962828640\junit1349019563279628173 (mmap=false) 16:43:04.768 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedIntermediateByte 16:43:04.768 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 16:43:04.768 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 16:43:04.999 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x0e541f25]' 16:43:04.999 DEBUG [main] StandbyServer.java:249 Binding was successful 16:43:05.036 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 11f59cf7-443c-4c82-8598-1f2109e54a54 for current head 16:43:05.052 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61945 is allowed 16:43:05.052 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 16:43:05.052 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 11f59cf7-443c-4c82-8598-1f2109e54a54 16:43:05.052 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 11f59cf7-443c-4c82-8598-1f2109e54a54 16:43:05.052 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 2b28eecd-75ac-43d9-a384-e4068fa544df.0000000c to client 11f59cf7-443c-4c82-8598-1f2109e54a54 16:43:05.052 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 16:43:05.052 DEBUG [standby-run-37] StandbyClientSyncExecution.java:103 Found missing segment 2b28eecd-75ac-43d9-a384-e4068fa544df 16:43:05.052 DEBUG [standby-run-37] StandbyClientSyncExecution.java:124 Inspecting segment 2b28eecd-75ac-43d9-a384-e4068fa544df 16:43:05.052 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 11f59cf7-443c-4c82-8598-1f2109e54a54 for references of segment 2b28eecd-75ac-43d9-a384-e4068fa544df 16:43:05.052 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 16:43:05.052 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 2b28eecd-75ac-43d9-a384-e4068fa544df for client 11f59cf7-443c-4c82-8598-1f2109e54a54 16:43:05.068 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 2b28eecd-75ac-43d9-a384-e4068fa544df to client 11f59cf7-443c-4c82-8598-1f2109e54a54 16:43:05.068 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 11f59cf7-443c-4c82-8598-1f2109e54a54 io.netty.handler.codec.compression.DecompressionException: mismatching checksum: a2826321 (expected: 464f41da) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 16:43:10.071 DEBUG [standby-run-37] StandbyClient.java:157 Channel closed 16:43:10.071 ERROR [standby-run-37] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment 2b28eecd-75ac-43d9-a384-e4068fa544df from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:346) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByte(DataStoreTestBase.java:311) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 16:43:12.288 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:43:12.342 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 for current head 16:43:12.342 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:61970 is allowed 16:43:12.342 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 16:43:12.342 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.342 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.342 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 2b28eecd-75ac-43d9-a384-e4068fa544df.0000000c to client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.342 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 16:43:12.342 DEBUG [standby-run-38] StandbyClientSyncExecution.java:103 Found missing segment 2b28eecd-75ac-43d9-a384-e4068fa544df 16:43:12.342 DEBUG [standby-run-38] StandbyClientSyncExecution.java:124 Inspecting segment 2b28eecd-75ac-43d9-a384-e4068fa544df 16:43:12.342 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 for references of segment 2b28eecd-75ac-43d9-a384-e4068fa544df 16:43:12.342 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 16:43:12.342 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 2b28eecd-75ac-43d9-a384-e4068fa544df for client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.342 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 2b28eecd-75ac-43d9-a384-e4068fa544df to client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.342 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:43:12.342 DEBUG [standby-run-38] StandbyClientSyncExecution.java:184 Found reference from 2b28eecd-75ac-43d9-a384-e4068fa544df to 10321473-d3fb-45d7-a853-c856c5c673b9 16:43:12.342 DEBUG [standby-run-38] StandbyClientSyncExecution.java:124 Inspecting segment 10321473-d3fb-45d7-a853-c856c5c673b9 16:43:12.342 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 for references of segment 10321473-d3fb-45d7-a853-c856c5c673b9 16:43:12.342 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 16:43:12.342 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 10321473-d3fb-45d7-a853-c856c5c673b9 for client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.342 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 10321473-d3fb-45d7-a853-c856c5c673b9 to client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.342 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:43:12.342 INFO [standby-run-38] StandbyClientSyncExecution.java:196 Copying data segment 10321473-d3fb-45d7-a853-c856c5c673b9 from primary 16:43:12.342 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 for segment 10321473-d3fb-45d7-a853-c856c5c673b9 16:43:12.342 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 16:43:12.342 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.342 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 10321473-d3fb-45d7-a853-c856c5c673b9 for client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.342 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.342 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 10321473-d3fb-45d7-a853-c856c5c673b9 to client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.358 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:43:12.358 INFO [standby-run-38] StandbyClientSyncExecution.java:196 Copying data segment 2b28eecd-75ac-43d9-a384-e4068fa544df from primary 16:43:12.358 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 for segment 2b28eecd-75ac-43d9-a384-e4068fa544df 16:43:12.358 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 16:43:12.358 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.358 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 2b28eecd-75ac-43d9-a384-e4068fa544df for client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.358 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.358 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 2b28eecd-75ac-43d9-a384-e4068fa544df to client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.358 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:43:12.358 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 for blob 4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880 16:43:12.358 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 16:43:12.358 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.358 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880 for client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.358 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 5242880 sent to client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.358 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880 to client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.358 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880 to client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.405 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:43:12.405 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880 16:43:12.405 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880.tmp 16:43:12.405 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880 to client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.442 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880 to client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.458 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:43:12.458 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880 16:43:12.458 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880.tmp 16:43:12.489 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:43:12.489 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880 16:43:12.489 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880 to client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.489 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880.tmp 16:43:12.526 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:43:12.526 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880 16:43:12.526 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880 to client 52ee8a81-6ace-4d9a-94f9-87cf0ad7a6f0 16:43:12.526 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880.tmp 16:43:12.558 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:43:12.558 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880 16:43:12.558 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880.tmp 16:43:12.573 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob 4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880 16:43:12.643 DEBUG [standby-run-38] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\4ed890d2de5c4d8b2f0441a44662937b26b5775007b55d51e05d4c2f9f768008#5242880.tmp 16:43:12.727 INFO [standby-run-38] StandbyClientSyncExecution.java:82 updated head state successfully: true in 385ms. 16:43:12.727 DEBUG [standby-run-38] StandbyClient.java:157 Channel closed 16:43:14.946 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:43:14.946 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:43:14.946 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:43:14.946 DEBUG [main] StandbyServer.java:231 Boss group shut down 16:43:14.961 DEBUG [main] StandbyServer.java:237 Worker group shut down 16:43:14.977 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedIntermediateByte 16:43:15.008 INFO [main] FileStore.java:466 TarMK closed: target\junit6743701212962828640\junit1349019563279628173 16:43:15.046 INFO [main] FileStore.java:466 TarMK closed: target\junit6743701212962828640\junit4303984672364308294 16:43:15.061 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit4064528528561050637\junit1692678864253223475, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:43:15.061 INFO [main] FileStore.java:240 TarMK opened: target\junit4064528528561050637\junit1692678864253223475 (mmap=false) 16:43:15.077 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit4064528528561050637\junit8930108879680913210, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:43:15.077 INFO [main] FileStore.java:240 TarMK opened: target\junit4064528528561050637\junit8930108879680913210 (mmap=false) 16:43:15.077 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedIntermediateByte2 16:43:15.077 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 16:43:15.077 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 16:43:15.246 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xb49e5aa1]' 16:43:15.246 DEBUG [main] StandbyServer.java:249 Binding was successful 16:43:15.293 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 0f563c0f-349f-4de4-985f-6710dc3a9058 for current head 16:43:15.308 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:62058 is allowed 16:43:15.308 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 16:43:15.308 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.308 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.308 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 6692b6d2-9b07-4273-ad51-e024c15c5cd9.0000000c to client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.308 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 16:43:15.308 DEBUG [standby-run-39] StandbyClientSyncExecution.java:103 Found missing segment 6692b6d2-9b07-4273-ad51-e024c15c5cd9 16:43:15.308 DEBUG [standby-run-39] StandbyClientSyncExecution.java:124 Inspecting segment 6692b6d2-9b07-4273-ad51-e024c15c5cd9 16:43:15.308 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 0f563c0f-349f-4de4-985f-6710dc3a9058 for references of segment 6692b6d2-9b07-4273-ad51-e024c15c5cd9 16:43:15.308 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 16:43:15.308 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 6692b6d2-9b07-4273-ad51-e024c15c5cd9 for client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.308 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 6692b6d2-9b07-4273-ad51-e024c15c5cd9 to client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.308 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:43:15.308 DEBUG [standby-run-39] StandbyClientSyncExecution.java:184 Found reference from 6692b6d2-9b07-4273-ad51-e024c15c5cd9 to 46a7135b-4856-4280-a6db-20c122f265c8 16:43:15.308 DEBUG [standby-run-39] StandbyClientSyncExecution.java:124 Inspecting segment 46a7135b-4856-4280-a6db-20c122f265c8 16:43:15.308 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 0f563c0f-349f-4de4-985f-6710dc3a9058 for references of segment 46a7135b-4856-4280-a6db-20c122f265c8 16:43:15.308 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 16:43:15.308 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 46a7135b-4856-4280-a6db-20c122f265c8 for client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.308 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 46a7135b-4856-4280-a6db-20c122f265c8 to client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.308 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:43:15.308 INFO [standby-run-39] StandbyClientSyncExecution.java:196 Copying data segment 46a7135b-4856-4280-a6db-20c122f265c8 from primary 16:43:15.308 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 0f563c0f-349f-4de4-985f-6710dc3a9058 for segment 46a7135b-4856-4280-a6db-20c122f265c8 16:43:15.308 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 16:43:15.308 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.308 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 46a7135b-4856-4280-a6db-20c122f265c8 for client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.308 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.308 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 46a7135b-4856-4280-a6db-20c122f265c8 to client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.308 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:43:15.308 INFO [standby-run-39] StandbyClientSyncExecution.java:196 Copying data segment 6692b6d2-9b07-4273-ad51-e024c15c5cd9 from primary 16:43:15.308 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 0f563c0f-349f-4de4-985f-6710dc3a9058 for segment 6692b6d2-9b07-4273-ad51-e024c15c5cd9 16:43:15.308 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 16:43:15.308 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.308 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 6692b6d2-9b07-4273-ad51-e024c15c5cd9 for client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.308 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.308 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 6692b6d2-9b07-4273-ad51-e024c15c5cd9 to client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.324 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:43:15.325 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 0f563c0f-349f-4de4-985f-6710dc3a9058 for blob 53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880 16:43:15.326 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 16:43:15.326 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get blob id' received from client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.326 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob 53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880 for client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.327 DEBUG [primary-1] CommunicationObserver.java:130 Binary with size 5242880 sent to client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.327 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob 53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880 to client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.330 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880 to client 0f563c0f-349f-4de4-985f-6710dc3a9058 16:43:15.346 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 0f563c0f-349f-4de4-985f-6710dc3a9058 io.netty.handler.codec.compression.DecompressionException: mismatching checksum: a90b7a9e (expected: 8ea0ffb3) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 16:43:20.328 DEBUG [standby-run-39] StandbyClient.java:157 Channel closed 16:43:20.329 ERROR [standby-run-39] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to load remote blob 53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880 at /root/server/#testBlob in 5000ms. Please increase the timeout and try again. at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchAndStoreBlob(StandbyDiff.java:208) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:196) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:181) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:169) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeAdded(StandbyDiff.java:94) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeAdded(CancelableDiff.java:76) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:562) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:566) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:101) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:346) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByte2(DataStoreTestBase.java:316) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 16:43:22.565 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:43:22.596 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 289fde1e-1db0-4e41-aa15-3418adbf69c8 for current head 16:43:22.596 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:62083 is allowed 16:43:22.596 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 16:43:22.596 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 289fde1e-1db0-4e41-aa15-3418adbf69c8 16:43:22.596 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 289fde1e-1db0-4e41-aa15-3418adbf69c8 16:43:22.596 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 6692b6d2-9b07-4273-ad51-e024c15c5cd9.0000000c to client 289fde1e-1db0-4e41-aa15-3418adbf69c8 16:43:22.596 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 16:43:22.596 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 289fde1e-1db0-4e41-aa15-3418adbf69c8 for blob 53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880 16:43:22.596 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 16:43:22.596 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client 289fde1e-1db0-4e41-aa15-3418adbf69c8 16:43:22.596 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880 for client 289fde1e-1db0-4e41-aa15-3418adbf69c8 16:43:22.596 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 5242880 sent to client 289fde1e-1db0-4e41-aa15-3418adbf69c8 16:43:22.596 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880 to client 289fde1e-1db0-4e41-aa15-3418adbf69c8 16:43:22.596 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880 to client 289fde1e-1db0-4e41-aa15-3418adbf69c8 16:43:22.635 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:43:22.635 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880 16:43:22.635 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880.tmp 16:43:22.635 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880 to client 289fde1e-1db0-4e41-aa15-3418adbf69c8 16:43:22.666 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:43:22.666 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880 16:43:22.666 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880.tmp 16:43:22.666 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880 to client 289fde1e-1db0-4e41-aa15-3418adbf69c8 16:43:22.713 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:43:22.713 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880 16:43:22.713 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880.tmp 16:43:22.713 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880 to client 289fde1e-1db0-4e41-aa15-3418adbf69c8 16:43:22.735 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:43:22.735 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880 16:43:22.735 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880.tmp 16:43:22.735 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880 to client 289fde1e-1db0-4e41-aa15-3418adbf69c8 16:43:22.782 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:43:22.782 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880 16:43:22.782 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880.tmp 16:43:22.797 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob 53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880 16:43:22.851 DEBUG [standby-run-40] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\53078effad0e869fc840ec4eed3ff9471d70af6640b64d9dcc05e7899609de21#5242880.tmp 16:43:22.932 INFO [standby-run-40] StandbyClientSyncExecution.java:82 updated head state successfully: true in 336ms. 16:43:22.933 DEBUG [standby-run-40] StandbyClient.java:157 Channel closed 16:43:25.183 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:43:25.183 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:43:25.183 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:43:25.183 DEBUG [main] StandbyServer.java:231 Boss group shut down 16:43:25.199 DEBUG [main] StandbyServer.java:237 Worker group shut down 16:43:25.214 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedIntermediateByte2 16:43:25.237 INFO [main] FileStore.java:466 TarMK closed: target\junit4064528528561050637\junit8930108879680913210 16:43:25.268 INFO [main] FileStore.java:466 TarMK closed: target\junit4064528528561050637\junit1692678864253223475 16:43:25.299 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7641795874927875769\junit2544732638116658869, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:43:25.299 INFO [main] FileStore.java:240 TarMK opened: target\junit7641795874927875769\junit2544732638116658869 (mmap=false) 16:43:25.299 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7641795874927875769\junit6819192287984881317, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:43:25.299 INFO [main] FileStore.java:240 TarMK opened: target\junit7641795874927875769\junit6819192287984881317 (mmap=false) 16:43:25.299 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedStartByte 16:43:25.299 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 16:43:25.299 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 16:43:25.483 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xedfd3f5a]' 16:43:25.483 DEBUG [main] StandbyServer.java:249 Binding was successful 16:43:25.537 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 60aaa4e7-a716-4336-897a-30abb4628e9f for current head 16:43:25.552 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:62172 is allowed 16:43:25.552 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 16:43:25.552 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 60aaa4e7-a716-4336-897a-30abb4628e9f 16:43:25.552 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 60aaa4e7-a716-4336-897a-30abb4628e9f 16:43:25.552 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 51b03d05-a5f3-42c5-ae72-4a51b41a5266.0000000c to client 60aaa4e7-a716-4336-897a-30abb4628e9f 16:43:25.552 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 60aaa4e7-a716-4336-897a-30abb4628e9f io.netty.handler.codec.compression.DecompressionException: Received COMPRESSED_DATA tag before STREAM_IDENTIFIER at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:162) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 16:43:30.554 ERROR [standby-run-41] StandbyClientSyncExecution.java:67 Unable to fetch remote head 16:43:30.554 DEBUG [standby-run-41] StandbyClient.java:157 Channel closed 16:43:32.763 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:43:32.779 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 for current head 16:43:32.779 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:62204 is allowed 16:43:32.779 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 16:43:32.779 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.779 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.779 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 51b03d05-a5f3-42c5-ae72-4a51b41a5266.0000000c to client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.779 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 16:43:32.779 DEBUG [standby-run-42] StandbyClientSyncExecution.java:103 Found missing segment 51b03d05-a5f3-42c5-ae72-4a51b41a5266 16:43:32.779 DEBUG [standby-run-42] StandbyClientSyncExecution.java:124 Inspecting segment 51b03d05-a5f3-42c5-ae72-4a51b41a5266 16:43:32.779 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 for references of segment 51b03d05-a5f3-42c5-ae72-4a51b41a5266 16:43:32.779 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 16:43:32.779 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 51b03d05-a5f3-42c5-ae72-4a51b41a5266 for client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.779 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 51b03d05-a5f3-42c5-ae72-4a51b41a5266 to client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.779 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:43:32.779 DEBUG [standby-run-42] StandbyClientSyncExecution.java:184 Found reference from 51b03d05-a5f3-42c5-ae72-4a51b41a5266 to a74477e7-876b-4ef4-a3c9-e4f0f8909d30 16:43:32.779 DEBUG [standby-run-42] StandbyClientSyncExecution.java:124 Inspecting segment a74477e7-876b-4ef4-a3c9-e4f0f8909d30 16:43:32.779 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 for references of segment a74477e7-876b-4ef4-a3c9-e4f0f8909d30 16:43:32.779 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 16:43:32.779 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment a74477e7-876b-4ef4-a3c9-e4f0f8909d30 for client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.779 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment a74477e7-876b-4ef4-a3c9-e4f0f8909d30 to client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.779 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:43:32.779 INFO [standby-run-42] StandbyClientSyncExecution.java:196 Copying data segment a74477e7-876b-4ef4-a3c9-e4f0f8909d30 from primary 16:43:32.779 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 for segment a74477e7-876b-4ef4-a3c9-e4f0f8909d30 16:43:32.779 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 16:43:32.779 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.779 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment a74477e7-876b-4ef4-a3c9-e4f0f8909d30 for client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.779 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.779 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment a74477e7-876b-4ef4-a3c9-e4f0f8909d30 to client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.779 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:43:32.779 INFO [standby-run-42] StandbyClientSyncExecution.java:196 Copying data segment 51b03d05-a5f3-42c5-ae72-4a51b41a5266 from primary 16:43:32.779 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 for segment 51b03d05-a5f3-42c5-ae72-4a51b41a5266 16:43:32.779 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 16:43:32.779 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.779 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 51b03d05-a5f3-42c5-ae72-4a51b41a5266 for client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.779 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.779 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 51b03d05-a5f3-42c5-ae72-4a51b41a5266 to client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.779 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:43:32.795 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 for blob 7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880 16:43:32.795 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 16:43:32.795 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.795 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880 for client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.795 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 5242880 sent to client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.795 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880 to client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.795 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880 to client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.835 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:43:32.835 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880 16:43:32.835 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880.tmp 16:43:32.835 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880 to client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.867 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:43:32.867 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880 16:43:32.867 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880.tmp 16:43:32.867 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880 to client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.914 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:43:32.914 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880 16:43:32.914 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880.tmp 16:43:32.914 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880 to client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.951 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:43:32.951 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880 16:43:32.951 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880.tmp 16:43:32.951 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880 to client 45fc8892-885f-4da3-9af4-c59ecc77c4b3 16:43:32.998 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 16:43:32.998 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880 16:43:32.998 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880.tmp 16:43:32.998 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob 7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880 16:43:33.067 DEBUG [standby-run-42] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\7281c9ad66e10900e1de7894629829ffe6c6dfbd8288fc5b4c8b435a36748f08#5242880.tmp 16:43:33.152 INFO [standby-run-42] StandbyClientSyncExecution.java:82 updated head state successfully: true in 373ms. 16:43:33.152 DEBUG [standby-run-42] StandbyClient.java:157 Channel closed 16:43:35.373 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:43:35.373 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:43:35.373 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:43:35.373 DEBUG [main] StandbyServer.java:231 Boss group shut down 16:43:35.373 DEBUG [main] StandbyServer.java:237 Worker group shut down 16:43:35.373 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedStartByte 16:43:35.420 INFO [main] FileStore.java:466 TarMK closed: target\junit7641795874927875769\junit6819192287984881317 16:43:35.442 INFO [main] FileStore.java:466 TarMK closed: target\junit7641795874927875769\junit2544732638116658869 16:43:35.473 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit656287719236092738\junit5231364631509754521, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:43:35.473 INFO [main] FileStore.java:240 TarMK opened: target\junit656287719236092738\junit5231364631509754521 (mmap=false) 16:43:35.473 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit656287719236092738\junit3504561725568894944, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 16:43:35.473 INFO [main] FileStore.java:240 TarMK opened: target\junit656287719236092738\junit3504561725568894944 (mmap=false) 16:43:35.489 INFO [main] DataStoreTestBase.java:127 Test begin: testSyncBigBlob 16:43:35.489 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 16:43:35.489 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 16:44:05.444 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xd04f734a]' 16:44:05.444 DEBUG [main] StandbyServer.java:249 Binding was successful 16:44:05.444 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 83695469-c6a6-40b4-a60b-6daf7baa7727 for current head 16:44:05.444 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:62290 is allowed 16:44:05.444 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 16:44:05.444 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 83695469-c6a6-40b4-a60b-6daf7baa7727 16:44:05.444 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 83695469-c6a6-40b4-a60b-6daf7baa7727 16:44:05.702 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 589146da-346e-4a14-abbe-c0fae0fa95c0.00000005 to client 83695469-c6a6-40b4-a60b-6daf7baa7727 16:44:05.702 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 16:44:05.702 DEBUG [standby-run-43] StandbyClientSyncExecution.java:103 Found missing segment 589146da-346e-4a14-abbe-c0fae0fa95c0 16:44:05.702 DEBUG [standby-run-43] StandbyClientSyncExecution.java:124 Inspecting segment 589146da-346e-4a14-abbe-c0fae0fa95c0 16:44:05.702 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 83695469-c6a6-40b4-a60b-6daf7baa7727 for references of segment 589146da-346e-4a14-abbe-c0fae0fa95c0 16:44:05.702 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 16:44:05.702 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 589146da-346e-4a14-abbe-c0fae0fa95c0 for client 83695469-c6a6-40b4-a60b-6daf7baa7727 16:44:05.791 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 589146da-346e-4a14-abbe-c0fae0fa95c0 to client 83695469-c6a6-40b4-a60b-6daf7baa7727 16:44:05.791 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 16:44:05.791 INFO [standby-run-43] StandbyClientSyncExecution.java:196 Copying data segment 589146da-346e-4a14-abbe-c0fae0fa95c0 from primary 16:44:05.791 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 83695469-c6a6-40b4-a60b-6daf7baa7727 for segment 589146da-346e-4a14-abbe-c0fae0fa95c0 16:44:05.791 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 16:44:05.791 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 83695469-c6a6-40b4-a60b-6daf7baa7727 16:44:05.791 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 589146da-346e-4a14-abbe-c0fae0fa95c0 for client 83695469-c6a6-40b4-a60b-6daf7baa7727 16:44:05.791 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client 83695469-c6a6-40b4-a60b-6daf7baa7727 16:44:05.791 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 589146da-346e-4a14-abbe-c0fae0fa95c0 to client 83695469-c6a6-40b4-a60b-6daf7baa7727 16:44:05.791 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 16:44:06.360 INFO [standby-run-43] StandbyClientSyncExecution.java:82 updated head state successfully: true in 658ms. 16:44:06.360 DEBUG [standby-run-43] StandbyClient.java:157 Channel closed 16:44:08.591 DEBUG [main] StandbyClientSync.java:277 Group shut down 16:44:08.591 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:44:08.591 DEBUG [main] StandbyServer.java:220 Channel disconnected 16:44:08.591 DEBUG [main] StandbyServer.java:231 Boss group shut down 16:44:08.606 DEBUG [main] StandbyServer.java:237 Worker group shut down 16:44:08.606 INFO [main] DataStoreTestBase.java:132 Test end: testSyncBigBlob 16:44:08.675 INFO [main] FileStore.java:466 TarMK closed: target\junit656287719236092738\junit3504561725568894944 16:44:08.744 INFO [main] FileStore.java:466 TarMK closed: target\junit656287719236092738\junit5231364631509754521