14:22:55.756 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit3912482732971027778\junit3582473194679558334, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:22:55.759 INFO [main] FileStore.java:240 TarMK opened: target\junit3912482732971027778\junit3582473194679558334 (mmap=false) 14:22:55.770 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit3912482732971027778\junit936374467627958834, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:22:55.773 INFO [main] FileStore.java:240 TarMK opened: target\junit3912482732971027778\junit936374467627958834 (mmap=false) 14:22:55.777 INFO [main] DataStoreTestBase.java:127 Test begin: testSyncFailingDueToTooShortTimeout 14:22:55.778 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:22:55.778 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:22:55.898 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xd936d8f3]' 14:22:55.900 DEBUG [main] StandbyServer.java:249 Binding was successful 14:22:56.683 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 for current head 14:22:56.684 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61003 is allowed 14:22:56.684 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:22:56.685 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.686 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.686 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 488c65dc-78fb-4cdb-a762-785ff5a8a4b6.0000000c to client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.687 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:22:56.688 DEBUG [standby-run-23] StandbyClientSyncExecution.java:103 Found missing segment 488c65dc-78fb-4cdb-a762-785ff5a8a4b6 14:22:56.689 DEBUG [standby-run-23] StandbyClientSyncExecution.java:124 Inspecting segment 488c65dc-78fb-4cdb-a762-785ff5a8a4b6 14:22:56.689 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 for references of segment 488c65dc-78fb-4cdb-a762-785ff5a8a4b6 14:22:56.690 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:22:56.690 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 488c65dc-78fb-4cdb-a762-785ff5a8a4b6 for client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.691 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 488c65dc-78fb-4cdb-a762-785ff5a8a4b6 to client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.691 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:22:56.692 DEBUG [standby-run-23] StandbyClientSyncExecution.java:184 Found reference from 488c65dc-78fb-4cdb-a762-785ff5a8a4b6 to e69f822b-c819-44c4-a6f9-38c5f2385e50 14:22:56.692 DEBUG [standby-run-23] StandbyClientSyncExecution.java:124 Inspecting segment e69f822b-c819-44c4-a6f9-38c5f2385e50 14:22:56.693 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 for references of segment e69f822b-c819-44c4-a6f9-38c5f2385e50 14:22:56.694 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:22:56.694 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment e69f822b-c819-44c4-a6f9-38c5f2385e50 for client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.694 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment e69f822b-c819-44c4-a6f9-38c5f2385e50 to client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.695 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:22:56.697 INFO [standby-run-23] StandbyClientSyncExecution.java:196 Copying data segment e69f822b-c819-44c4-a6f9-38c5f2385e50 from primary 14:22:56.698 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 for segment e69f822b-c819-44c4-a6f9-38c5f2385e50 14:22:56.698 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:22:56.698 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.699 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment e69f822b-c819-44c4-a6f9-38c5f2385e50 for client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.699 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.699 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment e69f822b-c819-44c4-a6f9-38c5f2385e50 to client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.700 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:22:56.700 INFO [standby-run-23] StandbyClientSyncExecution.java:196 Copying data segment 488c65dc-78fb-4cdb-a762-785ff5a8a4b6 from primary 14:22:56.701 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 for segment 488c65dc-78fb-4cdb-a762-785ff5a8a4b6 14:22:56.701 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:22:56.702 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.702 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 488c65dc-78fb-4cdb-a762-785ff5a8a4b6 for client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.702 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.702 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 488c65dc-78fb-4cdb-a762-785ff5a8a4b6 to client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.703 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:22:56.706 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 for blob 9c604a729ad3b6be81d715abcdca0b8a40e29e6e102d56e745d144529da036e3#5242880 14:22:56.708 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 14:22:56.708 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get blob id' received from client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.708 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob 9c604a729ad3b6be81d715abcdca0b8a40e29e6e102d56e745d144529da036e3#5242880 for client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.709 DEBUG [primary-1] CommunicationObserver.java:130 Binary with size 5242880 sent to client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.711 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob 9c604a729ad3b6be81d715abcdca0b8a40e29e6e102d56e745d144529da036e3#5242880 to client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.730 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 9c604a729ad3b6be81d715abcdca0b8a40e29e6e102d56e745d144529da036e3#5242880 to client 4cb7db75-9eb8-4df1-b632-0713aa0ee8a3 14:22:56.775 DEBUG [standby-run-23] StandbyClient.java:157 Channel closed 14:22:56.776 ERROR [standby-run-23] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to load remote blob 9c604a729ad3b6be81d715abcdca0b8a40e29e6e102d56e745d144529da036e3#5242880 at /root/server/#testBlob in 60ms. Please increase the timeout and try again. at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchAndStoreBlob(StandbyDiff.java:208) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:196) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:181) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:169) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeAdded(StandbyDiff.java:94) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeAdded(CancelableDiff.java:76) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:562) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:566) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:101) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.ExternalPrivateStoreIT.testSyncFailingDueToTooShortTimeout(ExternalPrivateStoreIT.java:86) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 14:22:58.981 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:22:58.981 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:22:58.982 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:22:58.983 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:22:58.990 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:22:58.990 INFO [main] DataStoreTestBase.java:132 Test end: testSyncFailingDueToTooShortTimeout 14:22:59.616 INFO [main] FileStore.java:466 TarMK closed: target\junit3912482732971027778\junit936374467627958834 14:22:59.652 INFO [main] FileStore.java:466 TarMK closed: target\junit3912482732971027778\junit3582473194679558334 14:22:59.672 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit4232757192096716441\junit1494994600031554794, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:22:59.676 INFO [main] FileStore.java:240 TarMK opened: target\junit4232757192096716441\junit1494994600031554794 (mmap=false) 14:22:59.688 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit4232757192096716441\junit2290696084889621165, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:22:59.691 INFO [main] FileStore.java:240 TarMK opened: target\junit4232757192096716441\junit2290696084889621165 (mmap=false) 14:22:59.694 INFO [main] DataStoreTestBase.java:127 Test begin: testResilientSync 14:22:59.695 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:22:59.695 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:22:59.803 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x4b14a5fd]' 14:22:59.805 DEBUG [main] StandbyServer.java:249 Binding was successful 14:22:59.808 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 986f4a76-d5e4-49f0-a4a7-fdbcd3ffd03e for current head 14:22:59.808 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61043 is allowed 14:22:59.809 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:22:59.809 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 986f4a76-d5e4-49f0-a4a7-fdbcd3ffd03e 14:22:59.810 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 986f4a76-d5e4-49f0-a4a7-fdbcd3ffd03e 14:23:03.808 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 986f4a76-d5e4-49f0-a4a7-fdbcd3ffd03e io.netty.handler.timeout.ReadTimeoutException: null 14:23:03.809 ERROR [standby-run-24] StandbyClientSyncExecution.java:67 Unable to fetch remote head 14:23:03.809 DEBUG [standby-run-24] StandbyClient.java:157 Channel closed 14:23:05.577 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 05460121-4651-482a-a779-7c9455eda20f.0000000c to client 986f4a76-d5e4-49f0-a4a7-fdbcd3ffd03e 14:23:05.581 WARN [primary-1] ExceptionHandler.java:31 Exception caught on the server java.io.IOException: Eine bestehende Verbindung wurde softwaregesteuert durch den Hostcomputer abgebrochen at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[na:1.8.0_151] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) ~[na:1.8.0_151] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_151] at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_151] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_151] at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288) ~[netty-buffer-4.1.14.Final.jar:4.1.14.Final] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106) ~[netty-buffer-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:372) ~[netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) ~[netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 14:23:06.013 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:23:06.013 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:23:06.013 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:23:06.014 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:23:06.016 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:23:06.042 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x9779222b]' 14:23:06.044 DEBUG [main] StandbyServer.java:249 Binding was successful 14:23:06.046 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 07b68798-0e71-40ad-8ad5-db9f52fc7798 for current head 14:23:06.047 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61085 is allowed 14:23:06.048 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:23:06.048 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.048 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.048 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 05460121-4651-482a-a779-7c9455eda20f.0000000c to client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.050 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:23:06.051 DEBUG [standby-run-25] StandbyClientSyncExecution.java:103 Found missing segment 05460121-4651-482a-a779-7c9455eda20f 14:23:06.051 DEBUG [standby-run-25] StandbyClientSyncExecution.java:124 Inspecting segment 05460121-4651-482a-a779-7c9455eda20f 14:23:06.051 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 07b68798-0e71-40ad-8ad5-db9f52fc7798 for references of segment 05460121-4651-482a-a779-7c9455eda20f 14:23:06.053 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:23:06.053 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 05460121-4651-482a-a779-7c9455eda20f for client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.054 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 05460121-4651-482a-a779-7c9455eda20f to client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.055 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:23:06.056 DEBUG [standby-run-25] StandbyClientSyncExecution.java:184 Found reference from 05460121-4651-482a-a779-7c9455eda20f to 88498fb6-9066-4b74-a5ef-b180932263d2 14:23:06.056 DEBUG [standby-run-25] StandbyClientSyncExecution.java:124 Inspecting segment 88498fb6-9066-4b74-a5ef-b180932263d2 14:23:06.056 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 07b68798-0e71-40ad-8ad5-db9f52fc7798 for references of segment 88498fb6-9066-4b74-a5ef-b180932263d2 14:23:06.057 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:23:06.057 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 88498fb6-9066-4b74-a5ef-b180932263d2 for client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.057 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 88498fb6-9066-4b74-a5ef-b180932263d2 to client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.058 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:23:06.058 INFO [standby-run-25] StandbyClientSyncExecution.java:196 Copying data segment 88498fb6-9066-4b74-a5ef-b180932263d2 from primary 14:23:06.058 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 07b68798-0e71-40ad-8ad5-db9f52fc7798 for segment 88498fb6-9066-4b74-a5ef-b180932263d2 14:23:06.059 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:23:06.059 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.059 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 88498fb6-9066-4b74-a5ef-b180932263d2 for client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.059 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.060 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 88498fb6-9066-4b74-a5ef-b180932263d2 to client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.060 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:23:06.061 INFO [standby-run-25] StandbyClientSyncExecution.java:196 Copying data segment 05460121-4651-482a-a779-7c9455eda20f from primary 14:23:06.061 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 07b68798-0e71-40ad-8ad5-db9f52fc7798 for segment 05460121-4651-482a-a779-7c9455eda20f 14:23:06.062 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:23:06.062 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.062 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 05460121-4651-482a-a779-7c9455eda20f for client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.062 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.063 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 05460121-4651-482a-a779-7c9455eda20f to client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.063 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:23:06.065 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 07b68798-0e71-40ad-8ad5-db9f52fc7798 for blob e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880 14:23:06.066 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 14:23:06.067 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get blob id' received from client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.067 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880 for client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.068 DEBUG [primary-1] CommunicationObserver.java:130 Binary with size 5242880 sent to client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.068 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880 to client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.073 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880 to client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.123 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:06.124 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880 14:23:06.133 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880 to client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.141 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880.tmp 14:23:06.196 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:06.197 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880 14:23:06.199 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880.tmp 14:23:06.200 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880 to client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.234 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:06.235 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880 14:23:06.236 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880 to client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.237 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880.tmp 14:23:06.274 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:06.274 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880 14:23:06.277 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880 to client 07b68798-0e71-40ad-8ad5-db9f52fc7798 14:23:06.277 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880.tmp 14:23:06.313 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:06.313 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880 14:23:06.315 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880.tmp 14:23:06.320 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880 14:23:06.378 DEBUG [standby-run-25] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\e09466c2611a53cbc81762f01c35730b60e0351a4e5f0a938f0f5a5c656b416a#5242880.tmp 14:23:07.291 INFO [standby-run-25] StandbyClientSyncExecution.java:82 updated head state successfully: true in 1240ms. 14:23:07.291 DEBUG [standby-run-25] StandbyClient.java:157 Channel closed 14:23:09.505 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:23:09.505 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:23:09.505 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:23:09.505 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:23:09.505 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:23:09.522 INFO [main] DataStoreTestBase.java:132 Test end: testResilientSync 14:23:09.567 INFO [main] FileStore.java:466 TarMK closed: target\junit4232757192096716441\junit2290696084889621165 14:23:09.602 INFO [main] FileStore.java:466 TarMK closed: target\junit4232757192096716441\junit1494994600031554794 14:23:09.626 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit8620242273635443082\junit6639862238981135586, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:23:09.630 INFO [main] FileStore.java:240 TarMK opened: target\junit8620242273635443082\junit6639862238981135586 (mmap=false) 14:23:09.640 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit8620242273635443082\junit5929049867773481302, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:23:09.644 INFO [main] FileStore.java:240 TarMK opened: target\junit8620242273635443082\junit5929049867773481302 (mmap=false) 14:23:09.657 INFO [main] DataStoreTestBase.java:127 Test begin: testProxySkippedBytes 14:23:09.658 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:23:09.658 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:23:09.765 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x4811f189]' 14:23:09.767 DEBUG [main] StandbyServer.java:249 Binding was successful 14:23:09.795 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=-1, skip=100,1 14:23:09.800 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 61102 14:23:10.903 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 for current head 14:23:10.918 DEBUG [proxy-worker-97] ForwardHandler.java:88 Connected to remote host 14:23:10.920 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61187 is allowed 14:23:10.920 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:23:10.921 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:10.921 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:10.922 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head cd27378c-3c08-4c53-aa74-4aae9d95c30f.0000000c to client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:10.924 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:23:10.924 DEBUG [standby-run-26] StandbyClientSyncExecution.java:103 Found missing segment cd27378c-3c08-4c53-aa74-4aae9d95c30f 14:23:10.924 DEBUG [standby-run-26] StandbyClientSyncExecution.java:124 Inspecting segment cd27378c-3c08-4c53-aa74-4aae9d95c30f 14:23:10.925 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 for references of segment cd27378c-3c08-4c53-aa74-4aae9d95c30f 14:23:10.926 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:23:10.926 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment cd27378c-3c08-4c53-aa74-4aae9d95c30f for client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:10.927 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment cd27378c-3c08-4c53-aa74-4aae9d95c30f to client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.926 DEBUG [standby-run-26] StandbyClient.java:157 Channel closed 14:23:15.926 DEBUG [proxy-worker-97] ForwardHandler.java:98 Connection to remote host closed 14:23:15.927 ERROR [standby-run-26] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment cd27378c-3c08-4c53-aa74-4aae9d95c30f from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:344) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxySkippedBytes(DataStoreTestBase.java:296) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 14:23:15.928 DEBUG [main] NetworkErrorProxy.java:116 Channel disconnected 14:23:15.929 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=-1, skip=-1,0 14:23:15.941 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 61102 14:23:15.941 DEBUG [proxy-worker-97] ForwardHandler.java:103 Group shut down 14:23:15.943 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 for current head 14:23:15.964 DEBUG [proxy-worker-98] ForwardHandler.java:88 Connected to remote host 14:23:15.965 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:61209 is allowed 14:23:15.966 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 14:23:15.966 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.967 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.967 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head cd27378c-3c08-4c53-aa74-4aae9d95c30f.0000000c to client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.968 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 14:23:15.969 DEBUG [standby-run-27] StandbyClientSyncExecution.java:103 Found missing segment cd27378c-3c08-4c53-aa74-4aae9d95c30f 14:23:15.969 DEBUG [standby-run-27] StandbyClientSyncExecution.java:124 Inspecting segment cd27378c-3c08-4c53-aa74-4aae9d95c30f 14:23:15.969 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 for references of segment cd27378c-3c08-4c53-aa74-4aae9d95c30f 14:23:15.970 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:23:15.970 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment cd27378c-3c08-4c53-aa74-4aae9d95c30f for client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.971 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment cd27378c-3c08-4c53-aa74-4aae9d95c30f to client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.972 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:23:15.973 DEBUG [standby-run-27] StandbyClientSyncExecution.java:184 Found reference from cd27378c-3c08-4c53-aa74-4aae9d95c30f to a5b7cd42-fef6-478f-a26e-cba2a49bf7ef 14:23:15.973 DEBUG [standby-run-27] StandbyClientSyncExecution.java:124 Inspecting segment a5b7cd42-fef6-478f-a26e-cba2a49bf7ef 14:23:15.973 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 for references of segment a5b7cd42-fef6-478f-a26e-cba2a49bf7ef 14:23:15.974 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:23:15.974 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment a5b7cd42-fef6-478f-a26e-cba2a49bf7ef for client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.975 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment a5b7cd42-fef6-478f-a26e-cba2a49bf7ef to client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.975 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:23:15.976 INFO [standby-run-27] StandbyClientSyncExecution.java:196 Copying data segment a5b7cd42-fef6-478f-a26e-cba2a49bf7ef from primary 14:23:15.976 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 for segment a5b7cd42-fef6-478f-a26e-cba2a49bf7ef 14:23:15.976 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:23:15.976 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.977 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment a5b7cd42-fef6-478f-a26e-cba2a49bf7ef for client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.977 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.977 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment a5b7cd42-fef6-478f-a26e-cba2a49bf7ef to client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.978 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:23:15.978 INFO [standby-run-27] StandbyClientSyncExecution.java:196 Copying data segment cd27378c-3c08-4c53-aa74-4aae9d95c30f from primary 14:23:15.979 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 for segment cd27378c-3c08-4c53-aa74-4aae9d95c30f 14:23:15.981 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:23:15.981 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.982 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment cd27378c-3c08-4c53-aa74-4aae9d95c30f for client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.983 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.983 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment cd27378c-3c08-4c53-aa74-4aae9d95c30f to client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.984 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:23:15.986 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 for blob bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880 14:23:15.987 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 14:23:15.987 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.987 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880 for client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.988 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 5242880 sent to client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.988 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880 to client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:15.994 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880 to client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:16.051 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880 to client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:16.087 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:16.088 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880 14:23:16.105 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880 to client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:16.112 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880.tmp 14:23:16.160 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880 to client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:16.203 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:16.203 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880 14:23:16.208 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880.tmp 14:23:16.298 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880 to client b8aa2f51-3bb2-40c6-bc3f-33e5a5fef1e8 14:23:16.401 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:16.402 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880 14:23:16.405 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880.tmp 14:23:16.487 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:16.487 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880 14:23:16.494 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880.tmp 14:23:16.528 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:16.529 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880 14:23:16.538 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880.tmp 14:23:16.549 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880 14:23:16.650 DEBUG [standby-run-27] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\bcd0d7ab637e54f5fd872ee9474472223fd27c24832dad4f27a3ac64e67dbfa9#5242880.tmp 14:23:17.840 INFO [standby-run-27] StandbyClientSyncExecution.java:82 updated head state successfully: true in 1871ms. 14:23:17.841 DEBUG [standby-run-27] StandbyClient.java:157 Channel closed 14:23:17.841 DEBUG [proxy-worker-98] ForwardHandler.java:98 Connection to remote host closed 14:23:17.847 DEBUG [proxy-worker-98] ForwardHandler.java:103 Group shut down 14:23:20.144 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:23:20.145 DEBUG [main] NetworkErrorProxy.java:128 Channel closed 14:23:20.152 DEBUG [main] NetworkErrorProxy.java:134 Boss group shut down 14:23:20.154 DEBUG [main] NetworkErrorProxy.java:139 Worker group shut down 14:23:20.155 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:23:20.155 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:23:20.156 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:23:20.158 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:23:20.174 INFO [main] DataStoreTestBase.java:132 Test end: testProxySkippedBytes 14:23:20.218 INFO [main] FileStore.java:466 TarMK closed: target\junit8620242273635443082\junit5929049867773481302 14:23:20.271 INFO [main] FileStore.java:466 TarMK closed: target\junit8620242273635443082\junit6639862238981135586 14:23:20.291 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit2366444227871550593\junit5936678278903274207, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:23:20.293 INFO [main] FileStore.java:240 TarMK opened: target\junit2366444227871550593\junit5936678278903274207 (mmap=false) 14:23:20.300 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit2366444227871550593\junit6589799507431684009, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:23:20.302 INFO [main] FileStore.java:240 TarMK opened: target\junit2366444227871550593\junit6589799507431684009 (mmap=false) 14:23:20.303 INFO [main] DataStoreTestBase.java:127 Test begin: testSyncUpdatedBinaryProperty 14:23:20.303 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:23:20.303 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:23:20.331 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x41e31533]' 14:23:20.333 DEBUG [main] StandbyServer.java:249 Binding was successful 14:23:21.049 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client da3d56f3-2beb-47d0-8cf9-27b312c3506a for current head 14:23:21.049 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61253 is allowed 14:23:21.050 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:23:21.050 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.051 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.051 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head a1b13d1e-b89a-49d7-ae00-14be57aaf9c1.0000000c to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.052 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:23:21.052 DEBUG [standby-run-28] StandbyClientSyncExecution.java:103 Found missing segment a1b13d1e-b89a-49d7-ae00-14be57aaf9c1 14:23:21.052 DEBUG [standby-run-28] StandbyClientSyncExecution.java:124 Inspecting segment a1b13d1e-b89a-49d7-ae00-14be57aaf9c1 14:23:21.053 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client da3d56f3-2beb-47d0-8cf9-27b312c3506a for references of segment a1b13d1e-b89a-49d7-ae00-14be57aaf9c1 14:23:21.054 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:23:21.054 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment a1b13d1e-b89a-49d7-ae00-14be57aaf9c1 for client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.055 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment a1b13d1e-b89a-49d7-ae00-14be57aaf9c1 to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.056 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:23:21.057 DEBUG [standby-run-28] StandbyClientSyncExecution.java:184 Found reference from a1b13d1e-b89a-49d7-ae00-14be57aaf9c1 to 943aa9d3-69f5-4a4a-af86-af3fc46987d1 14:23:21.057 DEBUG [standby-run-28] StandbyClientSyncExecution.java:124 Inspecting segment 943aa9d3-69f5-4a4a-af86-af3fc46987d1 14:23:21.058 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client da3d56f3-2beb-47d0-8cf9-27b312c3506a for references of segment 943aa9d3-69f5-4a4a-af86-af3fc46987d1 14:23:21.058 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:23:21.059 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 943aa9d3-69f5-4a4a-af86-af3fc46987d1 for client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.059 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 943aa9d3-69f5-4a4a-af86-af3fc46987d1 to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.060 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:23:21.060 INFO [standby-run-28] StandbyClientSyncExecution.java:196 Copying data segment 943aa9d3-69f5-4a4a-af86-af3fc46987d1 from primary 14:23:21.060 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client da3d56f3-2beb-47d0-8cf9-27b312c3506a for segment 943aa9d3-69f5-4a4a-af86-af3fc46987d1 14:23:21.061 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:23:21.062 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.062 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 943aa9d3-69f5-4a4a-af86-af3fc46987d1 for client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.062 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.062 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 943aa9d3-69f5-4a4a-af86-af3fc46987d1 to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.063 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:23:21.064 INFO [standby-run-28] StandbyClientSyncExecution.java:196 Copying data segment a1b13d1e-b89a-49d7-ae00-14be57aaf9c1 from primary 14:23:21.064 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client da3d56f3-2beb-47d0-8cf9-27b312c3506a for segment a1b13d1e-b89a-49d7-ae00-14be57aaf9c1 14:23:21.065 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:23:21.065 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.065 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment a1b13d1e-b89a-49d7-ae00-14be57aaf9c1 for client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.066 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.066 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment a1b13d1e-b89a-49d7-ae00-14be57aaf9c1 to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.066 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:23:21.068 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client da3d56f3-2beb-47d0-8cf9-27b312c3506a for blob 700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880 14:23:21.069 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 14:23:21.069 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get blob id' received from client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.069 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob 700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880 for client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.093 DEBUG [primary-1] CommunicationObserver.java:130 Binary with size 5242880 sent to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.094 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob 700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880 to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.096 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880 to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.143 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:21.143 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880 14:23:21.143 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880.tmp 14:23:21.143 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880 to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.182 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:21.182 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880 14:23:21.182 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880 to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.182 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880.tmp 14:23:21.229 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:21.229 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880 14:23:21.229 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880 to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.229 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880.tmp 14:23:21.260 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:21.260 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880 14:23:21.260 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880.tmp 14:23:21.260 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880 to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:21.297 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:21.297 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880 14:23:21.313 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880.tmp 14:23:21.313 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob 700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880 14:23:21.381 DEBUG [standby-run-28] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\700779ea16a3ca50f05b08d93e62b34e3f29fa9483c60bded3df5174e6f84b93#5242880.tmp 14:23:22.349 INFO [standby-run-28] StandbyClientSyncExecution.java:82 updated head state successfully: true in 1297ms. 14:23:22.350 DEBUG [standby-run-28] StandbyClient.java:157 Channel closed 14:23:23.350 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client da3d56f3-2beb-47d0-8cf9-27b312c3506a for current head 14:23:23.351 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:61256 is allowed 14:23:23.351 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 14:23:23.352 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:23.352 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:23.353 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 3598ab10-fdb8-4f65-a37b-fa0d2a063149.00000006 to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:23.353 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 14:23:23.354 DEBUG [standby-run-29] StandbyClientSyncExecution.java:103 Found missing segment 3598ab10-fdb8-4f65-a37b-fa0d2a063149 14:23:23.354 DEBUG [standby-run-29] StandbyClientSyncExecution.java:124 Inspecting segment 3598ab10-fdb8-4f65-a37b-fa0d2a063149 14:23:23.354 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client da3d56f3-2beb-47d0-8cf9-27b312c3506a for references of segment 3598ab10-fdb8-4f65-a37b-fa0d2a063149 14:23:23.355 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:23:23.355 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 3598ab10-fdb8-4f65-a37b-fa0d2a063149 for client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:23.355 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 3598ab10-fdb8-4f65-a37b-fa0d2a063149 to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:23.355 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:23:23.356 INFO [standby-run-29] StandbyClientSyncExecution.java:196 Copying data segment 3598ab10-fdb8-4f65-a37b-fa0d2a063149 from primary 14:23:23.356 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client da3d56f3-2beb-47d0-8cf9-27b312c3506a for segment 3598ab10-fdb8-4f65-a37b-fa0d2a063149 14:23:23.356 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:23:23.357 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:23.357 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 3598ab10-fdb8-4f65-a37b-fa0d2a063149 for client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:23.357 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 352 sent to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:23.357 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 3598ab10-fdb8-4f65-a37b-fa0d2a063149 to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:23.358 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:23:23.359 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client da3d56f3-2beb-47d0-8cf9-27b312c3506a for blob e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880 14:23:23.359 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 14:23:23.359 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:23.359 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880 for client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:23.360 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 5242880 sent to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:23.360 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880 to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:23.367 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880 to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:23.426 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:23.426 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880 14:23:23.431 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880.tmp 14:23:23.433 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880 to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:23.483 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:23.483 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880 14:23:23.486 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880.tmp 14:23:23.488 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880 to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:23.540 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:23.540 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880 14:23:23.543 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880.tmp 14:23:23.552 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880 to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:23.594 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:23.595 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880 14:23:23.598 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880 to client da3d56f3-2beb-47d0-8cf9-27b312c3506a 14:23:23.598 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880.tmp 14:23:23.670 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:24.043 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880 14:23:24.046 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880.tmp 14:23:24.074 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880 14:23:24.159 DEBUG [standby-run-29] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\e47e80f816bfd1f645c828439f5764c9048f248541bd988a3e8a178fc5bc4c28#5242880.tmp 14:23:24.925 INFO [standby-run-29] StandbyClientSyncExecution.java:82 updated head state successfully: true in 1571ms. 14:23:24.925 DEBUG [standby-run-29] StandbyClient.java:157 Channel closed 14:23:27.156 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:23:27.158 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:23:27.159 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:23:27.161 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:23:27.167 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:23:27.168 INFO [main] DataStoreTestBase.java:132 Test end: testSyncUpdatedBinaryProperty 14:23:27.210 INFO [main] FileStore.java:466 TarMK closed: target\junit2366444227871550593\junit6589799507431684009 14:23:27.227 INFO [main] FileStore.java:466 TarMK closed: target\junit2366444227871550593\junit5936678278903274207 14:23:27.259 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit5597258984404635454\junit9013910962884721886, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:23:27.259 INFO [main] FileStore.java:240 TarMK opened: target\junit5597258984404635454\junit9013910962884721886 (mmap=false) 14:23:27.500 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit5597258984404635454\junit6150756936150844404, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:23:27.501 INFO [main] FileStore.java:240 TarMK opened: target\junit5597258984404635454\junit6150756936150844404 (mmap=false) 14:23:27.563 INFO [main] DataStoreTestBase.java:127 Test begin: testSync 14:23:27.563 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:23:27.563 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:23:27.679 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x7ee91e2c]' 14:23:27.679 DEBUG [main] StandbyServer.java:249 Binding was successful 14:23:28.783 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client c6a0d24a-8615-49c2-a2ee-50786cc01835 for current head 14:23:28.784 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61300 is allowed 14:23:28.784 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:23:28.785 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.785 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.786 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head aadc927a-a111-4229-ae6f-823173511b51.0000000c to client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.787 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:23:28.788 DEBUG [standby-run-30] StandbyClientSyncExecution.java:103 Found missing segment aadc927a-a111-4229-ae6f-823173511b51 14:23:28.788 DEBUG [standby-run-30] StandbyClientSyncExecution.java:124 Inspecting segment aadc927a-a111-4229-ae6f-823173511b51 14:23:28.789 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client c6a0d24a-8615-49c2-a2ee-50786cc01835 for references of segment aadc927a-a111-4229-ae6f-823173511b51 14:23:28.790 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:23:28.790 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment aadc927a-a111-4229-ae6f-823173511b51 for client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.791 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment aadc927a-a111-4229-ae6f-823173511b51 to client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.791 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:23:28.792 DEBUG [standby-run-30] StandbyClientSyncExecution.java:184 Found reference from aadc927a-a111-4229-ae6f-823173511b51 to 17e9cde4-8702-4bae-a3bd-d8e5678d6ab5 14:23:28.792 DEBUG [standby-run-30] StandbyClientSyncExecution.java:124 Inspecting segment 17e9cde4-8702-4bae-a3bd-d8e5678d6ab5 14:23:28.800 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client c6a0d24a-8615-49c2-a2ee-50786cc01835 for references of segment 17e9cde4-8702-4bae-a3bd-d8e5678d6ab5 14:23:28.800 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:23:28.800 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 17e9cde4-8702-4bae-a3bd-d8e5678d6ab5 for client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.801 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 17e9cde4-8702-4bae-a3bd-d8e5678d6ab5 to client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.801 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:23:28.802 INFO [standby-run-30] StandbyClientSyncExecution.java:196 Copying data segment 17e9cde4-8702-4bae-a3bd-d8e5678d6ab5 from primary 14:23:28.802 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client c6a0d24a-8615-49c2-a2ee-50786cc01835 for segment 17e9cde4-8702-4bae-a3bd-d8e5678d6ab5 14:23:28.803 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:23:28.803 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.804 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 17e9cde4-8702-4bae-a3bd-d8e5678d6ab5 for client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.804 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.805 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 17e9cde4-8702-4bae-a3bd-d8e5678d6ab5 to client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.805 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:23:28.806 INFO [standby-run-30] StandbyClientSyncExecution.java:196 Copying data segment aadc927a-a111-4229-ae6f-823173511b51 from primary 14:23:28.806 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client c6a0d24a-8615-49c2-a2ee-50786cc01835 for segment aadc927a-a111-4229-ae6f-823173511b51 14:23:28.807 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:23:28.807 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.807 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment aadc927a-a111-4229-ae6f-823173511b51 for client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.807 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.808 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment aadc927a-a111-4229-ae6f-823173511b51 to client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.809 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:23:28.815 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client c6a0d24a-8615-49c2-a2ee-50786cc01835 for blob a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880 14:23:28.816 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 14:23:28.817 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get blob id' received from client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.817 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880 for client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.820 DEBUG [primary-1] CommunicationObserver.java:130 Binary with size 5242880 sent to client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.820 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880 to client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.827 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880 to client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.896 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:28.897 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880 14:23:28.901 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880.tmp 14:23:28.928 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880 to client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:28.945 ERROR [standby-1] Slf4JLogger.java:176 LEAK: ByteBuf.release() was not called before it's garbage-collected. Enable advanced leak reporting to find out where the leak occurred. To enable advanced leak reporting, specify the JVM option '-Dio.netty.leakDetection.level=advanced' or call ResourceLeakDetector.setLevel() See http://netty.io/wiki/reference-counted-objects.html for more information. 14:23:28.975 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:28.976 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880 14:23:28.980 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880.tmp 14:23:28.981 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880 to client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:29.030 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:29.030 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880 14:23:29.033 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880.tmp 14:23:29.034 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880 to client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:29.078 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:29.078 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880 14:23:29.082 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880.tmp 14:23:29.084 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880 to client c6a0d24a-8615-49c2-a2ee-50786cc01835 14:23:29.140 DEBUG [standby-1] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:29.503 DEBUG [standby-1] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880 14:23:29.505 DEBUG [standby-1] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880.tmp 14:23:29.676 DEBUG [standby-1] ResponseDecoder.java:167 Received entire blob a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880 14:23:29.730 DEBUG [standby-run-30] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\a0d1c27a83f36d8a54c0ac9320751cbbe1a8c7d0ece380176dc02f911112c70e#5242880.tmp 14:23:30.178 INFO [standby-run-30] StandbyClientSyncExecution.java:82 updated head state successfully: true in 1390ms. 14:23:30.178 DEBUG [standby-run-30] StandbyClient.java:157 Channel closed 14:23:32.393 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:23:32.393 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:23:32.393 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:23:32.393 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:23:32.393 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:23:32.408 INFO [main] DataStoreTestBase.java:132 Test end: testSync 14:23:32.439 INFO [main] FileStore.java:466 TarMK closed: target\junit5597258984404635454\junit6150756936150844404 14:23:32.477 INFO [main] FileStore.java:466 TarMK closed: target\junit5597258984404635454\junit9013910962884721886 14:23:32.493 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7060038712458248337\junit1618391802819657847, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:23:32.493 INFO [main] FileStore.java:240 TarMK opened: target\junit7060038712458248337\junit1618391802819657847 (mmap=false) 14:23:32.640 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7060038712458248337\junit2659582511699931803, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:23:32.640 INFO [main] FileStore.java:240 TarMK opened: target\junit7060038712458248337\junit2659582511699931803 (mmap=false) 14:23:32.662 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedIntermediateByteChange2 14:23:32.662 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:23:32.662 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:23:32.761 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xa3e5f271]' 14:23:32.761 DEBUG [main] StandbyServer.java:249 Binding was successful 14:23:32.809 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=150000, skip=0,0 14:23:32.809 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 61306 14:23:34.285 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client be41f7cf-d609-40a0-8bbf-4a22185a4fbd for current head 14:23:34.308 DEBUG [proxy-worker-105] ForwardHandler.java:88 Connected to remote host 14:23:34.309 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61394 is allowed 14:23:34.309 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:23:34.310 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.310 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.310 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head e9035e11-048c-4caf-ac29-b30e7ed9bcff.0000000c to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.311 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:23:34.312 DEBUG [standby-run-31] StandbyClientSyncExecution.java:103 Found missing segment e9035e11-048c-4caf-ac29-b30e7ed9bcff 14:23:34.312 DEBUG [standby-run-31] StandbyClientSyncExecution.java:124 Inspecting segment e9035e11-048c-4caf-ac29-b30e7ed9bcff 14:23:34.312 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client be41f7cf-d609-40a0-8bbf-4a22185a4fbd for references of segment e9035e11-048c-4caf-ac29-b30e7ed9bcff 14:23:34.314 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:23:34.314 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment e9035e11-048c-4caf-ac29-b30e7ed9bcff for client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.315 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment e9035e11-048c-4caf-ac29-b30e7ed9bcff to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.315 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:23:34.316 DEBUG [standby-run-31] StandbyClientSyncExecution.java:184 Found reference from e9035e11-048c-4caf-ac29-b30e7ed9bcff to a48595c0-4973-4064-abb6-0efc3061169f 14:23:34.316 DEBUG [standby-run-31] StandbyClientSyncExecution.java:124 Inspecting segment a48595c0-4973-4064-abb6-0efc3061169f 14:23:34.316 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client be41f7cf-d609-40a0-8bbf-4a22185a4fbd for references of segment a48595c0-4973-4064-abb6-0efc3061169f 14:23:34.317 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:23:34.317 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment a48595c0-4973-4064-abb6-0efc3061169f for client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.318 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment a48595c0-4973-4064-abb6-0efc3061169f to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.318 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:23:34.318 INFO [standby-run-31] StandbyClientSyncExecution.java:196 Copying data segment a48595c0-4973-4064-abb6-0efc3061169f from primary 14:23:34.319 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client be41f7cf-d609-40a0-8bbf-4a22185a4fbd for segment a48595c0-4973-4064-abb6-0efc3061169f 14:23:34.320 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:23:34.320 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.320 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment a48595c0-4973-4064-abb6-0efc3061169f for client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.320 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.320 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment a48595c0-4973-4064-abb6-0efc3061169f to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.321 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:23:34.322 INFO [standby-run-31] StandbyClientSyncExecution.java:196 Copying data segment e9035e11-048c-4caf-ac29-b30e7ed9bcff from primary 14:23:34.322 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client be41f7cf-d609-40a0-8bbf-4a22185a4fbd for segment e9035e11-048c-4caf-ac29-b30e7ed9bcff 14:23:34.323 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:23:34.323 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.324 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment e9035e11-048c-4caf-ac29-b30e7ed9bcff for client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.324 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.324 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment e9035e11-048c-4caf-ac29-b30e7ed9bcff to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.325 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:23:34.327 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client be41f7cf-d609-40a0-8bbf-4a22185a4fbd for blob 066391230e92e52ccf5e68729b0865b994d2f5ab14f7ae94797bd5b161a64657#5242880 14:23:34.328 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 14:23:34.328 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get blob id' received from client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.328 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob 066391230e92e52ccf5e68729b0865b994d2f5ab14f7ae94797bd5b161a64657#5242880 for client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.329 DEBUG [primary-1] CommunicationObserver.java:130 Binary with size 5242880 sent to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.329 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob 066391230e92e52ccf5e68729b0865b994d2f5ab14f7ae94797bd5b161a64657#5242880 to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.336 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 066391230e92e52ccf5e68729b0865b994d2f5ab14f7ae94797bd5b161a64657#5242880 to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.366 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client be41f7cf-d609-40a0-8bbf-4a22185a4fbd io.netty.handler.codec.compression.DecompressionException: mismatching checksum: a2820072 (expected: a2823cae) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 14:23:34.389 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 066391230e92e52ccf5e68729b0865b994d2f5ab14f7ae94797bd5b161a64657#5242880 to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.428 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 066391230e92e52ccf5e68729b0865b994d2f5ab14f7ae94797bd5b161a64657#5242880 to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.472 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 066391230e92e52ccf5e68729b0865b994d2f5ab14f7ae94797bd5b161a64657#5242880 to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:34.524 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 066391230e92e52ccf5e68729b0865b994d2f5ab14f7ae94797bd5b161a64657#5242880 to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:39.333 DEBUG [standby-run-31] StandbyClient.java:157 Channel closed 14:23:39.334 DEBUG [proxy-worker-105] ForwardHandler.java:98 Connection to remote host closed 14:23:39.335 ERROR [standby-run-31] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to load remote blob 066391230e92e52ccf5e68729b0865b994d2f5ab14f7ae94797bd5b161a64657#5242880 at /root/server/#testBlob in 5000ms. Please increase the timeout and try again. at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchAndStoreBlob(StandbyDiff.java:208) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:196) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:181) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:169) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeAdded(StandbyDiff.java:94) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeAdded(CancelableDiff.java:76) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:562) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:566) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:101) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:344) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByteChange2(DataStoreTestBase.java:326) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 14:23:39.337 DEBUG [main] NetworkErrorProxy.java:116 Channel disconnected 14:23:39.338 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=-1, skip=-1,0 14:23:39.338 DEBUG [proxy-worker-105] ForwardHandler.java:103 Group shut down 14:23:39.338 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 61306 14:23:40.016 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client be41f7cf-d609-40a0-8bbf-4a22185a4fbd for current head 14:23:40.036 DEBUG [proxy-worker-106] ForwardHandler.java:88 Connected to remote host 14:23:40.037 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:61418 is allowed 14:23:40.038 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 14:23:40.038 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:40.038 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:40.038 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head a5e43fc8-d9c2-4346-ada2-fd2bd931dddf.00000006 to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:40.038 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 14:23:40.038 DEBUG [standby-run-32] StandbyClientSyncExecution.java:103 Found missing segment a5e43fc8-d9c2-4346-ada2-fd2bd931dddf 14:23:40.038 DEBUG [standby-run-32] StandbyClientSyncExecution.java:124 Inspecting segment a5e43fc8-d9c2-4346-ada2-fd2bd931dddf 14:23:40.038 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client be41f7cf-d609-40a0-8bbf-4a22185a4fbd for references of segment a5e43fc8-d9c2-4346-ada2-fd2bd931dddf 14:23:40.038 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:23:40.038 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment a5e43fc8-d9c2-4346-ada2-fd2bd931dddf for client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:40.038 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment a5e43fc8-d9c2-4346-ada2-fd2bd931dddf to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:40.038 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:23:40.038 INFO [standby-run-32] StandbyClientSyncExecution.java:196 Copying data segment a5e43fc8-d9c2-4346-ada2-fd2bd931dddf from primary 14:23:40.038 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client be41f7cf-d609-40a0-8bbf-4a22185a4fbd for segment a5e43fc8-d9c2-4346-ada2-fd2bd931dddf 14:23:40.038 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:23:40.038 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:40.038 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment a5e43fc8-d9c2-4346-ada2-fd2bd931dddf for client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:40.038 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 352 sent to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:40.038 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment a5e43fc8-d9c2-4346-ada2-fd2bd931dddf to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:40.038 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:23:40.038 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client be41f7cf-d609-40a0-8bbf-4a22185a4fbd for blob 7f14a53c87a351edbef9ca1ac0531ae3cd18e52143f788670f8f735a7343a0dd#2097152 14:23:40.038 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 14:23:40.038 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:40.038 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 7f14a53c87a351edbef9ca1ac0531ae3cd18e52143f788670f8f735a7343a0dd#2097152 for client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:40.054 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 2097152 sent to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:40.054 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 7f14a53c87a351edbef9ca1ac0531ae3cd18e52143f788670f8f735a7343a0dd#2097152 to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:40.054 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/2 of size 1048576 from blob 7f14a53c87a351edbef9ca1ac0531ae3cd18e52143f788670f8f735a7343a0dd#2097152 to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:40.101 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/2 of size 1048576 from blob 7f14a53c87a351edbef9ca1ac0531ae3cd18e52143f788670f8f735a7343a0dd#2097152 to client be41f7cf-d609-40a0-8bbf-4a22185a4fbd 14:23:40.170 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:40.170 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/2 of size 1048576 from blob 7f14a53c87a351edbef9ca1ac0531ae3cd18e52143f788670f8f735a7343a0dd#2097152 14:23:40.170 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\7f14a53c87a351edbef9ca1ac0531ae3cd18e52143f788670f8f735a7343a0dd#2097152.tmp 14:23:40.217 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:40.217 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/2 of size 1048576 from blob 7f14a53c87a351edbef9ca1ac0531ae3cd18e52143f788670f8f735a7343a0dd#2097152 14:23:40.217 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\7f14a53c87a351edbef9ca1ac0531ae3cd18e52143f788670f8f735a7343a0dd#2097152.tmp 14:23:40.217 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob 7f14a53c87a351edbef9ca1ac0531ae3cd18e52143f788670f8f735a7343a0dd#2097152 14:23:40.239 DEBUG [standby-run-32] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\7f14a53c87a351edbef9ca1ac0531ae3cd18e52143f788670f8f735a7343a0dd#2097152.tmp 14:23:40.871 INFO [standby-run-32] StandbyClientSyncExecution.java:82 updated head state successfully: true in 833ms. 14:23:40.871 DEBUG [standby-run-32] StandbyClient.java:157 Channel closed 14:23:40.871 DEBUG [proxy-worker-106] ForwardHandler.java:98 Connection to remote host closed 14:23:40.871 DEBUG [proxy-worker-106] ForwardHandler.java:103 Group shut down 14:23:43.188 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:23:43.188 DEBUG [main] NetworkErrorProxy.java:128 Channel closed 14:23:43.188 DEBUG [main] NetworkErrorProxy.java:134 Boss group shut down 14:23:43.188 DEBUG [main] NetworkErrorProxy.java:139 Worker group shut down 14:23:43.188 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:23:43.188 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:23:43.188 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:23:43.188 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:23:43.204 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedIntermediateByteChange2 14:23:43.241 INFO [main] FileStore.java:466 TarMK closed: target\junit7060038712458248337\junit2659582511699931803 14:23:43.388 INFO [main] FileStore.java:466 TarMK closed: target\junit7060038712458248337\junit1618391802819657847 14:23:43.403 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit6750877676983104426\junit8389586314764353293, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:23:43.403 INFO [main] FileStore.java:240 TarMK opened: target\junit6750877676983104426\junit8389586314764353293 (mmap=false) 14:23:43.441 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit6750877676983104426\junit617706944842639838, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:23:43.441 INFO [main] FileStore.java:240 TarMK opened: target\junit6750877676983104426\junit617706944842639838 (mmap=false) 14:23:43.441 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedIntermediateByteChange 14:23:43.441 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:23:43.441 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:23:43.557 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x2866f0e2]' 14:23:43.557 DEBUG [main] StandbyServer.java:249 Binding was successful 14:23:43.573 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=150, skip=0,0 14:23:43.573 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 61423 14:23:44.772 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb for current head 14:23:44.804 DEBUG [proxy-worker-113] ForwardHandler.java:88 Connected to remote host 14:23:44.804 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61508 is allowed 14:23:44.819 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:23:44.819 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:44.819 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:44.819 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 0dd837eb-220e-4bdb-ad4d-b351e205d9d5.0000000c to client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:44.819 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:23:44.819 DEBUG [standby-run-33] StandbyClientSyncExecution.java:103 Found missing segment 0dd837eb-220e-4bdb-ad4d-b351e205d9d5 14:23:44.819 DEBUG [standby-run-33] StandbyClientSyncExecution.java:124 Inspecting segment 0dd837eb-220e-4bdb-ad4d-b351e205d9d5 14:23:44.819 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb for references of segment 0dd837eb-220e-4bdb-ad4d-b351e205d9d5 14:23:44.819 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:23:44.819 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 0dd837eb-220e-4bdb-ad4d-b351e205d9d5 for client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:44.819 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 0dd837eb-220e-4bdb-ad4d-b351e205d9d5 to client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:44.819 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb io.netty.handler.codec.compression.DecompressionException: mismatching checksum: 273bcf52 (expected: a282b609) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 14:23:49.838 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb io.netty.handler.timeout.ReadTimeoutException: null 14:23:49.839 DEBUG [standby-run-33] StandbyClient.java:157 Channel closed 14:23:49.840 DEBUG [proxy-worker-113] ForwardHandler.java:98 Connection to remote host closed 14:23:49.842 ERROR [standby-run-33] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment 0dd837eb-220e-4bdb-ad4d-b351e205d9d5 from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:344) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByteChange(DataStoreTestBase.java:321) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 14:23:49.844 DEBUG [main] NetworkErrorProxy.java:116 Channel disconnected 14:23:49.844 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=-1, skip=-1,0 14:23:49.844 DEBUG [proxy-worker-113] ForwardHandler.java:103 Group shut down 14:23:49.844 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 61423 14:23:50.359 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb for current head 14:23:50.390 DEBUG [proxy-worker-114] ForwardHandler.java:88 Connected to remote host 14:23:50.390 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:61532 is allowed 14:23:50.390 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 14:23:50.390 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.390 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.390 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head a3f5016d-a85c-40e5-a587-25c97d3fd016.00000006 to client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.390 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 14:23:50.390 DEBUG [standby-run-34] StandbyClientSyncExecution.java:103 Found missing segment a3f5016d-a85c-40e5-a587-25c97d3fd016 14:23:50.390 DEBUG [standby-run-34] StandbyClientSyncExecution.java:124 Inspecting segment a3f5016d-a85c-40e5-a587-25c97d3fd016 14:23:50.390 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb for references of segment a3f5016d-a85c-40e5-a587-25c97d3fd016 14:23:50.390 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:23:50.390 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment a3f5016d-a85c-40e5-a587-25c97d3fd016 for client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.390 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment a3f5016d-a85c-40e5-a587-25c97d3fd016 to client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.390 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:23:50.390 DEBUG [standby-run-34] StandbyClientSyncExecution.java:184 Found reference from a3f5016d-a85c-40e5-a587-25c97d3fd016 to 0dd837eb-220e-4bdb-ad4d-b351e205d9d5 14:23:50.390 DEBUG [standby-run-34] StandbyClientSyncExecution.java:184 Found reference from a3f5016d-a85c-40e5-a587-25c97d3fd016 to 6c780723-a5fa-49fe-a3a7-dca28c295d59 14:23:50.390 DEBUG [standby-run-34] StandbyClientSyncExecution.java:124 Inspecting segment 0dd837eb-220e-4bdb-ad4d-b351e205d9d5 14:23:50.390 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb for references of segment 0dd837eb-220e-4bdb-ad4d-b351e205d9d5 14:23:50.390 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:23:50.406 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 0dd837eb-220e-4bdb-ad4d-b351e205d9d5 for client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.406 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 0dd837eb-220e-4bdb-ad4d-b351e205d9d5 to client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.406 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:23:50.406 DEBUG [standby-run-34] StandbyClientSyncExecution.java:124 Inspecting segment 6c780723-a5fa-49fe-a3a7-dca28c295d59 14:23:50.406 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb for references of segment 6c780723-a5fa-49fe-a3a7-dca28c295d59 14:23:50.406 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:23:50.406 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 6c780723-a5fa-49fe-a3a7-dca28c295d59 for client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.406 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 6c780723-a5fa-49fe-a3a7-dca28c295d59 to client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.406 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:23:50.406 INFO [standby-run-34] StandbyClientSyncExecution.java:196 Copying data segment 6c780723-a5fa-49fe-a3a7-dca28c295d59 from primary 14:23:50.406 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb for segment 6c780723-a5fa-49fe-a3a7-dca28c295d59 14:23:50.406 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:23:50.406 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.406 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 6c780723-a5fa-49fe-a3a7-dca28c295d59 for client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.406 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.406 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 6c780723-a5fa-49fe-a3a7-dca28c295d59 to client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.406 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:23:50.406 INFO [standby-run-34] StandbyClientSyncExecution.java:196 Copying data segment 0dd837eb-220e-4bdb-ad4d-b351e205d9d5 from primary 14:23:50.406 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb for segment 0dd837eb-220e-4bdb-ad4d-b351e205d9d5 14:23:50.406 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:23:50.406 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.406 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 0dd837eb-220e-4bdb-ad4d-b351e205d9d5 for client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.406 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.406 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 0dd837eb-220e-4bdb-ad4d-b351e205d9d5 to client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.406 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:23:50.406 INFO [standby-run-34] StandbyClientSyncExecution.java:196 Copying data segment a3f5016d-a85c-40e5-a587-25c97d3fd016 from primary 14:23:50.406 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb for segment a3f5016d-a85c-40e5-a587-25c97d3fd016 14:23:50.406 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:23:50.406 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.406 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment a3f5016d-a85c-40e5-a587-25c97d3fd016 for client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.406 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 352 sent to client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.406 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment a3f5016d-a85c-40e5-a587-25c97d3fd016 to client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.421 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:23:50.421 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb for blob 44c6ffee8cb71ab8c94aa79d893af8127914d8658ea083d63f3514555a60972b#2097152 14:23:50.421 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 14:23:50.421 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.421 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 44c6ffee8cb71ab8c94aa79d893af8127914d8658ea083d63f3514555a60972b#2097152 for client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.421 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 2097152 sent to client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.421 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 44c6ffee8cb71ab8c94aa79d893af8127914d8658ea083d63f3514555a60972b#2097152 to client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.421 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/2 of size 1048576 from blob 44c6ffee8cb71ab8c94aa79d893af8127914d8658ea083d63f3514555a60972b#2097152 to client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.460 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/2 of size 1048576 from blob 44c6ffee8cb71ab8c94aa79d893af8127914d8658ea083d63f3514555a60972b#2097152 to client 0c6d99df-e17c-49bb-be53-0f81fc2b1feb 14:23:50.492 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:50.492 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/2 of size 1048576 from blob 44c6ffee8cb71ab8c94aa79d893af8127914d8658ea083d63f3514555a60972b#2097152 14:23:50.492 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\44c6ffee8cb71ab8c94aa79d893af8127914d8658ea083d63f3514555a60972b#2097152.tmp 14:23:50.577 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:23:50.577 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/2 of size 1048576 from blob 44c6ffee8cb71ab8c94aa79d893af8127914d8658ea083d63f3514555a60972b#2097152 14:23:50.580 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\44c6ffee8cb71ab8c94aa79d893af8127914d8658ea083d63f3514555a60972b#2097152.tmp 14:23:50.586 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob 44c6ffee8cb71ab8c94aa79d893af8127914d8658ea083d63f3514555a60972b#2097152 14:23:50.607 DEBUG [standby-run-34] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\44c6ffee8cb71ab8c94aa79d893af8127914d8658ea083d63f3514555a60972b#2097152.tmp 14:23:50.963 INFO [standby-run-34] StandbyClientSyncExecution.java:82 updated head state successfully: true in 573ms. 14:23:50.963 DEBUG [standby-run-34] StandbyClient.java:157 Channel closed 14:23:50.963 DEBUG [proxy-worker-114] ForwardHandler.java:98 Connection to remote host closed 14:23:50.963 DEBUG [proxy-worker-114] ForwardHandler.java:103 Group shut down 14:23:53.298 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:23:53.298 DEBUG [main] NetworkErrorProxy.java:128 Channel closed 14:23:53.298 DEBUG [main] NetworkErrorProxy.java:134 Boss group shut down 14:23:53.313 DEBUG [main] NetworkErrorProxy.java:139 Worker group shut down 14:23:53.313 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:23:53.313 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:23:53.313 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:23:53.329 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:23:53.329 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedIntermediateByteChange 14:23:53.382 INFO [main] FileStore.java:466 TarMK closed: target\junit6750877676983104426\junit617706944842639838 14:23:53.413 INFO [main] FileStore.java:466 TarMK closed: target\junit6750877676983104426\junit8389586314764353293 14:23:53.451 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit2494161372343631846\junit6260287056882177806, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:23:53.451 INFO [main] FileStore.java:240 TarMK opened: target\junit2494161372343631846\junit6260287056882177806 (mmap=false) 14:23:53.451 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit2494161372343631846\junit6375386509795710106, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:23:53.467 INFO [main] FileStore.java:240 TarMK opened: target\junit2494161372343631846\junit6375386509795710106 (mmap=false) 14:23:53.467 INFO [main] DataStoreTestBase.java:127 Test begin: testProxySkippedBytesIntermediateChange 14:23:53.467 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:23:53.467 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:23:53.567 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xeadb3c3b]' 14:23:53.567 DEBUG [main] StandbyServer.java:249 Binding was successful 14:23:53.598 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=-1, skip=100,1 14:23:53.598 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 61538 14:23:55.253 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 971b14e8-18bc-475d-914f-c78d79f4cd4a for current head 14:23:55.255 DEBUG [proxy-worker-121] ForwardHandler.java:88 Connected to remote host 14:23:55.255 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61627 is allowed 14:23:55.255 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:23:55.255 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:23:55.255 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:23:55.255 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 0f573a32-1d70-434f-aa82-03758d36bcee.0000000c to client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:23:55.255 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:23:55.271 DEBUG [standby-run-35] StandbyClientSyncExecution.java:103 Found missing segment 0f573a32-1d70-434f-aa82-03758d36bcee 14:23:55.271 DEBUG [standby-run-35] StandbyClientSyncExecution.java:124 Inspecting segment 0f573a32-1d70-434f-aa82-03758d36bcee 14:23:55.271 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 971b14e8-18bc-475d-914f-c78d79f4cd4a for references of segment 0f573a32-1d70-434f-aa82-03758d36bcee 14:23:55.271 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:23:55.271 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 0f573a32-1d70-434f-aa82-03758d36bcee for client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:23:55.271 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 0f573a32-1d70-434f-aa82-03758d36bcee to client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:00.274 DEBUG [standby-run-35] StandbyClient.java:157 Channel closed 14:24:00.275 DEBUG [proxy-worker-121] ForwardHandler.java:98 Connection to remote host closed 14:24:00.275 ERROR [standby-run-35] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment 0f573a32-1d70-434f-aa82-03758d36bcee from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:344) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxySkippedBytesIntermediateChange(DataStoreTestBase.java:301) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 14:24:00.276 DEBUG [main] NetworkErrorProxy.java:116 Channel disconnected 14:24:00.278 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=-1, skip=-1,0 14:24:00.278 DEBUG [proxy-worker-121] ForwardHandler.java:103 Group shut down 14:24:00.279 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 61538 14:24:01.209 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client 971b14e8-18bc-475d-914f-c78d79f4cd4a for current head 14:24:01.233 DEBUG [proxy-worker-122] ForwardHandler.java:88 Connected to remote host 14:24:01.235 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:61655 is allowed 14:24:01.235 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 14:24:01.235 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.236 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.236 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 10f343c7-dc88-4544-a1ba-467967865a5d.00000006 to client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.237 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 14:24:01.237 DEBUG [standby-run-36] StandbyClientSyncExecution.java:103 Found missing segment 10f343c7-dc88-4544-a1ba-467967865a5d 14:24:01.237 DEBUG [standby-run-36] StandbyClientSyncExecution.java:124 Inspecting segment 10f343c7-dc88-4544-a1ba-467967865a5d 14:24:01.238 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 971b14e8-18bc-475d-914f-c78d79f4cd4a for references of segment 10f343c7-dc88-4544-a1ba-467967865a5d 14:24:01.238 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:24:01.238 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 10f343c7-dc88-4544-a1ba-467967865a5d for client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.239 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 10f343c7-dc88-4544-a1ba-467967865a5d to client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.239 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:24:01.239 DEBUG [standby-run-36] StandbyClientSyncExecution.java:184 Found reference from 10f343c7-dc88-4544-a1ba-467967865a5d to 0f573a32-1d70-434f-aa82-03758d36bcee 14:24:01.240 DEBUG [standby-run-36] StandbyClientSyncExecution.java:184 Found reference from 10f343c7-dc88-4544-a1ba-467967865a5d to 1f1a9559-9395-4304-accc-da7d1bc0cced 14:24:01.240 DEBUG [standby-run-36] StandbyClientSyncExecution.java:124 Inspecting segment 0f573a32-1d70-434f-aa82-03758d36bcee 14:24:01.240 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 971b14e8-18bc-475d-914f-c78d79f4cd4a for references of segment 0f573a32-1d70-434f-aa82-03758d36bcee 14:24:01.241 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:24:01.241 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 0f573a32-1d70-434f-aa82-03758d36bcee for client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.242 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 0f573a32-1d70-434f-aa82-03758d36bcee to client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.242 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:24:01.243 DEBUG [standby-run-36] StandbyClientSyncExecution.java:124 Inspecting segment 1f1a9559-9395-4304-accc-da7d1bc0cced 14:24:01.243 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 971b14e8-18bc-475d-914f-c78d79f4cd4a for references of segment 1f1a9559-9395-4304-accc-da7d1bc0cced 14:24:01.244 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:24:01.244 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 1f1a9559-9395-4304-accc-da7d1bc0cced for client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.244 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 1f1a9559-9395-4304-accc-da7d1bc0cced to client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.245 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:24:01.245 INFO [standby-run-36] StandbyClientSyncExecution.java:196 Copying data segment 1f1a9559-9395-4304-accc-da7d1bc0cced from primary 14:24:01.246 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 971b14e8-18bc-475d-914f-c78d79f4cd4a for segment 1f1a9559-9395-4304-accc-da7d1bc0cced 14:24:01.249 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:24:01.249 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.249 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 1f1a9559-9395-4304-accc-da7d1bc0cced for client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.250 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.250 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 1f1a9559-9395-4304-accc-da7d1bc0cced to client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.251 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:24:01.252 INFO [standby-run-36] StandbyClientSyncExecution.java:196 Copying data segment 0f573a32-1d70-434f-aa82-03758d36bcee from primary 14:24:01.252 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 971b14e8-18bc-475d-914f-c78d79f4cd4a for segment 0f573a32-1d70-434f-aa82-03758d36bcee 14:24:01.252 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:24:01.252 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.253 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 0f573a32-1d70-434f-aa82-03758d36bcee for client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.253 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.253 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 0f573a32-1d70-434f-aa82-03758d36bcee to client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.254 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:24:01.254 INFO [standby-run-36] StandbyClientSyncExecution.java:196 Copying data segment 10f343c7-dc88-4544-a1ba-467967865a5d from primary 14:24:01.254 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 971b14e8-18bc-475d-914f-c78d79f4cd4a for segment 10f343c7-dc88-4544-a1ba-467967865a5d 14:24:01.255 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:24:01.255 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.255 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 10f343c7-dc88-4544-a1ba-467967865a5d for client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.255 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 352 sent to client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.255 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 10f343c7-dc88-4544-a1ba-467967865a5d to client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.256 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:24:01.257 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client 971b14e8-18bc-475d-914f-c78d79f4cd4a for blob b11018d99ec9a47cf7520bab08f2dc1bb6f0786a7aa58b477649032e1cd4d27c#2097152 14:24:01.257 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 14:24:01.257 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.257 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob b11018d99ec9a47cf7520bab08f2dc1bb6f0786a7aa58b477649032e1cd4d27c#2097152 for client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.258 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 2097152 sent to client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.258 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob b11018d99ec9a47cf7520bab08f2dc1bb6f0786a7aa58b477649032e1cd4d27c#2097152 to client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.261 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/2 of size 1048576 from blob b11018d99ec9a47cf7520bab08f2dc1bb6f0786a7aa58b477649032e1cd4d27c#2097152 to client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.310 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/2 of size 1048576 from blob b11018d99ec9a47cf7520bab08f2dc1bb6f0786a7aa58b477649032e1cd4d27c#2097152 to client 971b14e8-18bc-475d-914f-c78d79f4cd4a 14:24:01.333 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:24:01.334 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/2 of size 1048576 from blob b11018d99ec9a47cf7520bab08f2dc1bb6f0786a7aa58b477649032e1cd4d27c#2097152 14:24:01.337 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\b11018d99ec9a47cf7520bab08f2dc1bb6f0786a7aa58b477649032e1cd4d27c#2097152.tmp 14:24:01.401 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:24:01.404 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/2 of size 1048576 from blob b11018d99ec9a47cf7520bab08f2dc1bb6f0786a7aa58b477649032e1cd4d27c#2097152 14:24:01.407 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\b11018d99ec9a47cf7520bab08f2dc1bb6f0786a7aa58b477649032e1cd4d27c#2097152.tmp 14:24:01.414 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob b11018d99ec9a47cf7520bab08f2dc1bb6f0786a7aa58b477649032e1cd4d27c#2097152 14:24:01.442 DEBUG [standby-run-36] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\b11018d99ec9a47cf7520bab08f2dc1bb6f0786a7aa58b477649032e1cd4d27c#2097152.tmp 14:24:01.876 INFO [standby-run-36] StandbyClientSyncExecution.java:82 updated head state successfully: true in 639ms. 14:24:01.877 DEBUG [standby-run-36] StandbyClient.java:157 Channel closed 14:24:01.877 DEBUG [proxy-worker-122] ForwardHandler.java:98 Connection to remote host closed 14:24:01.886 DEBUG [proxy-worker-122] ForwardHandler.java:103 Group shut down 14:24:04.181 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:24:04.181 DEBUG [main] NetworkErrorProxy.java:128 Channel closed 14:24:04.181 DEBUG [main] NetworkErrorProxy.java:134 Boss group shut down 14:24:04.181 DEBUG [main] NetworkErrorProxy.java:139 Worker group shut down 14:24:04.181 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:24:04.181 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:24:04.181 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:24:04.181 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:24:04.197 INFO [main] DataStoreTestBase.java:132 Test end: testProxySkippedBytesIntermediateChange 14:24:04.228 INFO [main] FileStore.java:466 TarMK closed: target\junit2494161372343631846\junit6375386509795710106 14:24:04.250 INFO [main] FileStore.java:466 TarMK closed: target\junit2494161372343631846\junit6260287056882177806 14:24:04.281 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1764753359633119034\junit7475301422382454769, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:24:04.281 INFO [main] FileStore.java:240 TarMK opened: target\junit1764753359633119034\junit7475301422382454769 (mmap=false) 14:24:04.281 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1764753359633119034\junit4504651632164952707, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:24:04.281 INFO [main] FileStore.java:240 TarMK opened: target\junit1764753359633119034\junit4504651632164952707 (mmap=false) 14:24:04.281 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedIntermediateByte 14:24:04.297 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:24:04.297 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:24:04.382 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xa40b6f88]' 14:24:04.382 DEBUG [main] StandbyServer.java:249 Binding was successful 14:24:04.397 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=150, skip=0,0 14:24:04.397 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 61660 14:24:05.114 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 for current head 14:24:05.151 DEBUG [proxy-worker-129] ForwardHandler.java:88 Connected to remote host 14:24:05.151 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61745 is allowed 14:24:05.151 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:24:05.151 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:05.151 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:05.151 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head e51da868-98ad-460c-a093-74f9c420ccd6.0000000c to client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:05.151 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:24:05.151 DEBUG [standby-run-37] StandbyClientSyncExecution.java:103 Found missing segment e51da868-98ad-460c-a093-74f9c420ccd6 14:24:05.151 DEBUG [standby-run-37] StandbyClientSyncExecution.java:124 Inspecting segment e51da868-98ad-460c-a093-74f9c420ccd6 14:24:05.151 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 for references of segment e51da868-98ad-460c-a093-74f9c420ccd6 14:24:05.151 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:24:05.151 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment e51da868-98ad-460c-a093-74f9c420ccd6 for client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:05.151 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment e51da868-98ad-460c-a093-74f9c420ccd6 to client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:05.151 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 io.netty.handler.codec.compression.DecompressionException: mismatching checksum: 2727abe7 (expected: a282d91c) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 14:24:10.174 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 io.netty.handler.timeout.ReadTimeoutException: null 14:24:10.174 DEBUG [standby-run-37] StandbyClient.java:157 Channel closed 14:24:10.174 DEBUG [proxy-worker-129] ForwardHandler.java:98 Connection to remote host closed 14:24:10.174 ERROR [standby-run-37] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment e51da868-98ad-460c-a093-74f9c420ccd6 from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:344) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByte(DataStoreTestBase.java:311) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 14:24:10.174 DEBUG [main] NetworkErrorProxy.java:116 Channel disconnected 14:24:10.174 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=-1, skip=-1,0 14:24:10.174 DEBUG [proxy-worker-129] ForwardHandler.java:103 Group shut down 14:24:10.174 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 61660 14:24:10.174 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 for current head 14:24:10.189 DEBUG [proxy-worker-130] ForwardHandler.java:88 Connected to remote host 14:24:10.189 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:61767 is allowed 14:24:10.189 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 14:24:10.189 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.189 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.189 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head e51da868-98ad-460c-a093-74f9c420ccd6.0000000c to client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.189 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 14:24:10.189 DEBUG [standby-run-38] StandbyClientSyncExecution.java:103 Found missing segment e51da868-98ad-460c-a093-74f9c420ccd6 14:24:10.189 DEBUG [standby-run-38] StandbyClientSyncExecution.java:124 Inspecting segment e51da868-98ad-460c-a093-74f9c420ccd6 14:24:10.189 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 for references of segment e51da868-98ad-460c-a093-74f9c420ccd6 14:24:10.189 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:24:10.189 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment e51da868-98ad-460c-a093-74f9c420ccd6 for client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.189 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment e51da868-98ad-460c-a093-74f9c420ccd6 to client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.189 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:24:10.189 DEBUG [standby-run-38] StandbyClientSyncExecution.java:184 Found reference from e51da868-98ad-460c-a093-74f9c420ccd6 to 3bb9f26e-d326-43f8-a994-a67954b15da6 14:24:10.189 DEBUG [standby-run-38] StandbyClientSyncExecution.java:124 Inspecting segment 3bb9f26e-d326-43f8-a994-a67954b15da6 14:24:10.189 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 for references of segment 3bb9f26e-d326-43f8-a994-a67954b15da6 14:24:10.189 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:24:10.189 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 3bb9f26e-d326-43f8-a994-a67954b15da6 for client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.189 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 3bb9f26e-d326-43f8-a994-a67954b15da6 to client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.189 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:24:10.189 INFO [standby-run-38] StandbyClientSyncExecution.java:196 Copying data segment 3bb9f26e-d326-43f8-a994-a67954b15da6 from primary 14:24:10.189 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 for segment 3bb9f26e-d326-43f8-a994-a67954b15da6 14:24:10.189 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:24:10.205 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.205 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 3bb9f26e-d326-43f8-a994-a67954b15da6 for client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.205 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.205 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 3bb9f26e-d326-43f8-a994-a67954b15da6 to client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.205 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:24:10.205 INFO [standby-run-38] StandbyClientSyncExecution.java:196 Copying data segment e51da868-98ad-460c-a093-74f9c420ccd6 from primary 14:24:10.205 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 for segment e51da868-98ad-460c-a093-74f9c420ccd6 14:24:10.205 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:24:10.205 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.205 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment e51da868-98ad-460c-a093-74f9c420ccd6 for client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.205 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.205 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment e51da868-98ad-460c-a093-74f9c420ccd6 to client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.205 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:24:10.205 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 for blob 349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880 14:24:10.205 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 14:24:10.205 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.205 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880 for client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.205 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 5242880 sent to client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.205 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880 to client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.205 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880 to client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.256 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880 to client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.274 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880 to client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.289 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:24:10.289 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880 14:24:10.289 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880.tmp 14:24:10.321 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880 to client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.366 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880 to client c58205ce-b4e1-4e4f-b7eb-49a1b7a43315 14:24:10.403 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:24:10.403 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880 14:24:10.406 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880.tmp 14:24:10.476 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:24:10.477 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880 14:24:10.478 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880.tmp 14:24:10.527 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:24:10.527 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880 14:24:10.530 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880.tmp 14:24:10.545 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:24:10.546 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880 14:24:10.548 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880.tmp 14:24:10.553 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob 349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880 14:24:10.615 DEBUG [standby-run-38] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\349090403b816fd49499227853132ddbde42f215c1fe7ee0580964b219cd50a9#5242880.tmp 14:24:11.039 INFO [standby-run-38] StandbyClientSyncExecution.java:82 updated head state successfully: true in 850ms. 14:24:11.055 DEBUG [proxy-worker-130] ForwardHandler.java:98 Connection to remote host closed 14:24:11.055 DEBUG [standby-run-38] StandbyClient.java:157 Channel closed 14:24:11.070 DEBUG [proxy-worker-130] ForwardHandler.java:103 Group shut down 14:24:13.273 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:24:13.274 DEBUG [main] NetworkErrorProxy.java:128 Channel closed 14:24:13.276 DEBUG [main] NetworkErrorProxy.java:134 Boss group shut down 14:24:13.278 DEBUG [main] NetworkErrorProxy.java:139 Worker group shut down 14:24:13.279 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:24:13.279 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:24:13.282 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:24:13.286 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:24:13.295 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedIntermediateByte 14:24:13.354 INFO [main] FileStore.java:466 TarMK closed: target\junit1764753359633119034\junit4504651632164952707 14:24:13.386 INFO [main] FileStore.java:466 TarMK closed: target\junit1764753359633119034\junit7475301422382454769 14:24:13.406 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7305726311700712593\junit6193915075018944334, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:24:13.409 INFO [main] FileStore.java:240 TarMK opened: target\junit7305726311700712593\junit6193915075018944334 (mmap=false) 14:24:13.774 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7305726311700712593\junit1649304763791722008, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:24:13.778 INFO [main] FileStore.java:240 TarMK opened: target\junit7305726311700712593\junit1649304763791722008 (mmap=false) 14:24:13.889 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedIntermediateByte2 14:24:13.889 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:24:13.889 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:24:14.002 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xb4561257]' 14:24:14.004 DEBUG [main] StandbyServer.java:249 Binding was successful 14:24:14.028 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=150000, skip=0,0 14:24:14.034 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 61773 14:24:15.362 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 for current head 14:24:15.376 DEBUG [proxy-worker-137] ForwardHandler.java:88 Connected to remote host 14:24:15.377 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61860 is allowed 14:24:15.378 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:24:15.378 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.379 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.380 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head d08ff2a7-683a-45ac-a897-235dd3cd2d4e.0000000c to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.382 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:24:15.382 DEBUG [standby-run-39] StandbyClientSyncExecution.java:103 Found missing segment d08ff2a7-683a-45ac-a897-235dd3cd2d4e 14:24:15.383 DEBUG [standby-run-39] StandbyClientSyncExecution.java:124 Inspecting segment d08ff2a7-683a-45ac-a897-235dd3cd2d4e 14:24:15.383 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 for references of segment d08ff2a7-683a-45ac-a897-235dd3cd2d4e 14:24:15.384 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:24:15.384 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment d08ff2a7-683a-45ac-a897-235dd3cd2d4e for client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.385 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment d08ff2a7-683a-45ac-a897-235dd3cd2d4e to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.386 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:24:15.386 DEBUG [standby-run-39] StandbyClientSyncExecution.java:184 Found reference from d08ff2a7-683a-45ac-a897-235dd3cd2d4e to 8e7fdd0b-c06f-4422-a7ee-6096fa95c76e 14:24:15.386 DEBUG [standby-run-39] StandbyClientSyncExecution.java:124 Inspecting segment 8e7fdd0b-c06f-4422-a7ee-6096fa95c76e 14:24:15.386 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 for references of segment 8e7fdd0b-c06f-4422-a7ee-6096fa95c76e 14:24:15.387 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:24:15.388 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 8e7fdd0b-c06f-4422-a7ee-6096fa95c76e for client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.388 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 8e7fdd0b-c06f-4422-a7ee-6096fa95c76e to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.388 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:24:15.389 INFO [standby-run-39] StandbyClientSyncExecution.java:196 Copying data segment 8e7fdd0b-c06f-4422-a7ee-6096fa95c76e from primary 14:24:15.389 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 for segment 8e7fdd0b-c06f-4422-a7ee-6096fa95c76e 14:24:15.389 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:24:15.390 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.390 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 8e7fdd0b-c06f-4422-a7ee-6096fa95c76e for client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.390 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.390 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 8e7fdd0b-c06f-4422-a7ee-6096fa95c76e to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.391 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:24:15.391 INFO [standby-run-39] StandbyClientSyncExecution.java:196 Copying data segment d08ff2a7-683a-45ac-a897-235dd3cd2d4e from primary 14:24:15.391 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 for segment d08ff2a7-683a-45ac-a897-235dd3cd2d4e 14:24:15.392 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:24:15.392 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.392 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment d08ff2a7-683a-45ac-a897-235dd3cd2d4e for client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.392 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.392 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment d08ff2a7-683a-45ac-a897-235dd3cd2d4e to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.393 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:24:15.394 DEBUG [standby-1] GetBlobRequestEncoder.java:33 Sending request from client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 for blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 14:24:15.395 DEBUG [primary-1] RequestDecoder.java:39 Parsed 'get blob' request 14:24:15.395 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get blob id' received from client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.395 DEBUG [primary-1] GetBlobRequestHandler.java:41 Reading blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 for client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.396 DEBUG [primary-1] CommunicationObserver.java:130 Binary with size 5242880 sent to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.396 DEBUG [primary-1] GetBlobResponseEncoder.java:41 Sending blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.401 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.421 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 io.netty.handler.codec.compression.DecompressionException: mismatching checksum: a2829fce (expected: a2821ac3) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 14:24:15.446 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.511 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.527 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:15.573 DEBUG [primary-1] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:20.395 DEBUG [standby-run-39] StandbyClient.java:157 Channel closed 14:24:20.395 DEBUG [proxy-worker-137] ForwardHandler.java:98 Connection to remote host closed 14:24:20.396 ERROR [standby-run-39] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to load remote blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 at /root/server/#testBlob in 5000ms. Please increase the timeout and try again. at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchAndStoreBlob(StandbyDiff.java:208) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:196) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:181) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.fetchBinary(StandbyDiff.java:169) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeAdded(StandbyDiff.java:94) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.CancelableDiff.childNodeAdded(CancelableDiff.java:76) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:562) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.diff(StandbyDiff.java:143) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.process(StandbyDiff.java:121) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyDiff.childNodeChanged(StandbyDiff.java:105) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.SegmentNodeState.compareAgainstBaseState(SegmentNodeState.java:566) ~[oak-segment-tar-1.8-SNAPSHOT.jar:na] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:101) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:344) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByte2(DataStoreTestBase.java:316) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 14:24:20.398 DEBUG [main] NetworkErrorProxy.java:116 Channel disconnected 14:24:20.398 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=-1, skip=-1,0 14:24:20.399 DEBUG [proxy-worker-137] ForwardHandler.java:103 Group shut down 14:24:20.400 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 61773 14:24:20.402 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 for current head 14:24:20.413 DEBUG [proxy-worker-138] ForwardHandler.java:88 Connected to remote host 14:24:20.414 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:61883 is allowed 14:24:20.414 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 14:24:20.414 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:20.415 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:20.415 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head d08ff2a7-683a-45ac-a897-235dd3cd2d4e.0000000c to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:20.416 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 14:24:20.416 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 for blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 14:24:20.417 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 14:24:20.417 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:20.417 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 for client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:20.417 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 5242880 sent to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:20.418 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:20.420 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:20.453 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:20.491 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:24:20.491 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 14:24:20.493 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880.tmp 14:24:20.494 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:20.535 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:20.566 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 to client 7423e2f4-2a42-4c8e-b26a-d367c0d4ec60 14:24:20.599 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:24:20.599 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 14:24:20.601 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880.tmp 14:24:20.645 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:24:20.646 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 14:24:20.647 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880.tmp 14:24:20.676 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:24:20.676 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 14:24:20.679 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880.tmp 14:24:20.700 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:24:20.700 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 14:24:20.702 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880.tmp 14:24:20.709 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880 14:24:20.769 DEBUG [standby-run-40] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\a4eef5f84aaffc4418fcc20333c04541d308a32955d666d04715f94719a039e6#5242880.tmp 14:24:21.842 INFO [standby-run-40] StandbyClientSyncExecution.java:82 updated head state successfully: true in 1426ms. 14:24:21.842 DEBUG [standby-run-40] StandbyClient.java:157 Channel closed 14:24:21.842 DEBUG [proxy-worker-138] ForwardHandler.java:98 Connection to remote host closed 14:24:21.845 DEBUG [proxy-worker-138] ForwardHandler.java:103 Group shut down 14:24:24.144 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:24:24.146 DEBUG [main] NetworkErrorProxy.java:128 Channel closed 14:24:24.149 DEBUG [main] NetworkErrorProxy.java:134 Boss group shut down 14:24:24.151 DEBUG [main] NetworkErrorProxy.java:139 Worker group shut down 14:24:24.151 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:24:24.152 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:24:24.152 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:24:24.154 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:24:24.164 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedIntermediateByte2 14:24:24.195 INFO [main] FileStore.java:466 TarMK closed: target\junit7305726311700712593\junit1649304763791722008 14:24:24.223 INFO [main] FileStore.java:466 TarMK closed: target\junit7305726311700712593\junit6193915075018944334 14:24:24.244 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7512176347581529901\junit4739857388473020443, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:24:24.247 INFO [main] FileStore.java:240 TarMK opened: target\junit7512176347581529901\junit4739857388473020443 (mmap=false) 14:24:24.252 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit7512176347581529901\junit8733740929807246961, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:24:24.254 INFO [main] FileStore.java:240 TarMK opened: target\junit7512176347581529901\junit8733740929807246961 (mmap=false) 14:24:24.261 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedStartByte 14:24:24.261 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:24:24.261 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:24:24.351 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xc51733ba]' 14:24:24.353 DEBUG [main] StandbyServer.java:249 Binding was successful 14:24:24.374 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=0, skip=0,0 14:24:24.375 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 61887 14:24:25.597 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client c758d47c-9364-4540-b25b-380e1357fa6e for current head 14:24:25.618 DEBUG [proxy-worker-145] ForwardHandler.java:88 Connected to remote host 14:24:25.619 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:61973 is allowed 14:24:25.619 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:24:25.619 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:25.620 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:25.620 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head aee70b24-341a-4b4b-a077-599d1d941dfd.0000000c to client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:25.621 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client c758d47c-9364-4540-b25b-380e1357fa6e io.netty.handler.codec.compression.DecompressionException: Received COMPRESSED_DATA tag before STREAM_IDENTIFIER at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:162) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 14:24:30.597 ERROR [standby-run-41] StandbyClientSyncExecution.java:67 Unable to fetch remote head 14:24:30.598 DEBUG [standby-run-41] StandbyClient.java:157 Channel closed 14:24:30.598 DEBUG [proxy-worker-145] ForwardHandler.java:98 Connection to remote host closed 14:24:30.598 DEBUG [main] NetworkErrorProxy.java:116 Channel disconnected 14:24:30.599 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=-1, skip=-1,0 14:24:30.601 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 61887 14:24:30.602 DEBUG [proxy-worker-145] ForwardHandler.java:103 Group shut down 14:24:30.604 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client c758d47c-9364-4540-b25b-380e1357fa6e for current head 14:24:30.618 DEBUG [proxy-worker-146] ForwardHandler.java:88 Connected to remote host 14:24:30.619 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:61996 is allowed 14:24:30.619 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 14:24:30.619 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.620 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.620 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head aee70b24-341a-4b4b-a077-599d1d941dfd.0000000c to client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.621 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 14:24:30.621 DEBUG [standby-run-42] StandbyClientSyncExecution.java:103 Found missing segment aee70b24-341a-4b4b-a077-599d1d941dfd 14:24:30.621 DEBUG [standby-run-42] StandbyClientSyncExecution.java:124 Inspecting segment aee70b24-341a-4b4b-a077-599d1d941dfd 14:24:30.621 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client c758d47c-9364-4540-b25b-380e1357fa6e for references of segment aee70b24-341a-4b4b-a077-599d1d941dfd 14:24:30.622 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:24:30.622 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment aee70b24-341a-4b4b-a077-599d1d941dfd for client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.623 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment aee70b24-341a-4b4b-a077-599d1d941dfd to client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.624 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:24:30.624 DEBUG [standby-run-42] StandbyClientSyncExecution.java:184 Found reference from aee70b24-341a-4b4b-a077-599d1d941dfd to f4e8a525-2783-4d71-a223-c0990366dc91 14:24:30.624 DEBUG [standby-run-42] StandbyClientSyncExecution.java:124 Inspecting segment f4e8a525-2783-4d71-a223-c0990366dc91 14:24:30.625 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client c758d47c-9364-4540-b25b-380e1357fa6e for references of segment f4e8a525-2783-4d71-a223-c0990366dc91 14:24:30.626 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:24:30.626 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment f4e8a525-2783-4d71-a223-c0990366dc91 for client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.626 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment f4e8a525-2783-4d71-a223-c0990366dc91 to client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.627 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:24:30.627 INFO [standby-run-42] StandbyClientSyncExecution.java:196 Copying data segment f4e8a525-2783-4d71-a223-c0990366dc91 from primary 14:24:30.627 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client c758d47c-9364-4540-b25b-380e1357fa6e for segment f4e8a525-2783-4d71-a223-c0990366dc91 14:24:30.628 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:24:30.628 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.628 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment f4e8a525-2783-4d71-a223-c0990366dc91 for client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.629 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.629 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment f4e8a525-2783-4d71-a223-c0990366dc91 to client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.630 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:24:30.630 INFO [standby-run-42] StandbyClientSyncExecution.java:196 Copying data segment aee70b24-341a-4b4b-a077-599d1d941dfd from primary 14:24:30.631 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client c758d47c-9364-4540-b25b-380e1357fa6e for segment aee70b24-341a-4b4b-a077-599d1d941dfd 14:24:30.631 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:24:30.632 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.632 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment aee70b24-341a-4b4b-a077-599d1d941dfd for client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.632 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.632 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment aee70b24-341a-4b4b-a077-599d1d941dfd to client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.633 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:24:30.635 DEBUG [standby-2] GetBlobRequestEncoder.java:33 Sending request from client c758d47c-9364-4540-b25b-380e1357fa6e for blob 0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880 14:24:30.635 DEBUG [primary-2] RequestDecoder.java:39 Parsed 'get blob' request 14:24:30.636 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get blob id' received from client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.636 DEBUG [primary-2] GetBlobRequestHandler.java:41 Reading blob 0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880 for client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.636 DEBUG [primary-2] CommunicationObserver.java:130 Binary with size 5242880 sent to client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.637 DEBUG [primary-2] GetBlobResponseEncoder.java:41 Sending blob 0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880 to client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.640 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 1/5 of size 1048576 from blob 0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880 to client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.690 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 2/5 of size 1048576 from blob 0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880 to client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.718 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:24:30.718 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 1/5 of size 1048576 from blob 0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880 14:24:30.721 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880.tmp 14:24:30.738 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 3/5 of size 1048576 from blob 0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880 to client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.790 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 4/5 of size 1048576 from blob 0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880 to client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.834 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:24:30.834 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 2/5 of size 1048576 from blob 0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880 14:24:30.837 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880.tmp 14:24:30.839 DEBUG [primary-2] ChunkedBlobStream.java:128 Sending chunk 5/5 of size 1048576 from blob 0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880 to client c758d47c-9364-4540-b25b-380e1357fa6e 14:24:30.962 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:24:30.962 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 3/5 of size 1048576 from blob 0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880 14:24:30.964 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880.tmp 14:24:31.049 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:24:31.049 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 4/5 of size 1048576 from blob 0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880 14:24:31.052 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880.tmp 14:24:31.125 DEBUG [standby-2] ResponseDecoder.java:90 Decoding 'get blob' response 14:24:31.125 DEBUG [standby-2] ResponseDecoder.java:150 Received chunk 5/5 of size 1048576 from blob 0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880 14:24:31.127 DEBUG [standby-2] ResponseDecoder.java:159 All checks OK. Appending chunk to disk to C:\tmp\0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880.tmp 14:24:31.187 DEBUG [standby-2] ResponseDecoder.java:167 Received entire blob 0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880 14:24:31.246 DEBUG [standby-run-42] ResponseDecoder.java:66 Processing input stream finished! Deleting file C:\tmp\0456548e8e9bfe3d3a2ab10ea9625b366f9f7795bf30a3249b69a89a010aa881#5242880.tmp 14:24:31.996 INFO [standby-run-42] StandbyClientSyncExecution.java:82 updated head state successfully: true in 1375ms. 14:24:31.996 DEBUG [standby-run-42] StandbyClient.java:157 Channel closed 14:24:31.997 DEBUG [proxy-worker-146] ForwardHandler.java:98 Connection to remote host closed 14:24:32.002 DEBUG [proxy-worker-146] ForwardHandler.java:103 Group shut down 14:24:34.301 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:24:34.302 DEBUG [main] NetworkErrorProxy.java:128 Channel closed 14:24:34.306 DEBUG [main] NetworkErrorProxy.java:134 Boss group shut down 14:24:34.310 DEBUG [main] NetworkErrorProxy.java:139 Worker group shut down 14:24:34.310 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:24:34.311 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:24:34.311 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:24:34.316 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:24:34.328 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedStartByte 14:24:34.384 INFO [main] FileStore.java:466 TarMK closed: target\junit7512176347581529901\junit8733740929807246961 14:24:34.447 INFO [main] FileStore.java:466 TarMK closed: target\junit7512176347581529901\junit4739857388473020443 14:24:34.480 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1350584405308229636\junit7862800850442952006, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:24:34.486 INFO [main] FileStore.java:240 TarMK opened: target\junit1350584405308229636\junit7862800850442952006 (mmap=false) 14:24:35.151 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1350584405308229636\junit6739816403331306109, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:24:35.154 INFO [main] FileStore.java:240 TarMK opened: target\junit1350584405308229636\junit6739816403331306109 (mmap=false) 14:24:35.560 INFO [main] DataStoreTestBase.java:127 Test begin: testSyncBigBlob 14:24:35.560 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:24:35.560 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:25:19.213 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x65c96ffb]' 14:25:19.213 DEBUG [main] StandbyServer.java:249 Binding was successful 14:25:19.213 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 1a591615-cec6-41c5-87a3-4dcac6801129 for current head 14:25:19.213 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:62091 is allowed 14:25:19.213 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:25:19.213 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 1a591615-cec6-41c5-87a3-4dcac6801129 14:25:19.213 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 1a591615-cec6-41c5-87a3-4dcac6801129 14:25:19.728 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 1a4ecae9-e0df-4216-adc1-c5f7d48d897d.00000005 to client 1a591615-cec6-41c5-87a3-4dcac6801129 14:25:19.728 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:25:19.728 DEBUG [standby-run-43] StandbyClientSyncExecution.java:103 Found missing segment 1a4ecae9-e0df-4216-adc1-c5f7d48d897d 14:25:19.728 DEBUG [standby-run-43] StandbyClientSyncExecution.java:124 Inspecting segment 1a4ecae9-e0df-4216-adc1-c5f7d48d897d 14:25:19.728 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 1a591615-cec6-41c5-87a3-4dcac6801129 for references of segment 1a4ecae9-e0df-4216-adc1-c5f7d48d897d 14:25:19.728 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:25:19.728 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 1a4ecae9-e0df-4216-adc1-c5f7d48d897d for client 1a591615-cec6-41c5-87a3-4dcac6801129 14:25:19.928 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 1a4ecae9-e0df-4216-adc1-c5f7d48d897d to client 1a591615-cec6-41c5-87a3-4dcac6801129 14:25:19.928 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:25:19.928 INFO [standby-run-43] StandbyClientSyncExecution.java:196 Copying data segment 1a4ecae9-e0df-4216-adc1-c5f7d48d897d from primary 14:25:19.928 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 1a591615-cec6-41c5-87a3-4dcac6801129 for segment 1a4ecae9-e0df-4216-adc1-c5f7d48d897d 14:25:19.928 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:25:19.928 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 1a591615-cec6-41c5-87a3-4dcac6801129 14:25:19.928 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 1a4ecae9-e0df-4216-adc1-c5f7d48d897d for client 1a591615-cec6-41c5-87a3-4dcac6801129 14:25:19.928 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client 1a591615-cec6-41c5-87a3-4dcac6801129 14:25:19.928 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 1a4ecae9-e0df-4216-adc1-c5f7d48d897d to client 1a591615-cec6-41c5-87a3-4dcac6801129 14:25:19.928 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:25:20.831 INFO [standby-run-43] StandbyClientSyncExecution.java:82 updated head state successfully: true in 1103ms. 14:25:20.831 DEBUG [standby-run-43] StandbyClient.java:157 Channel closed 14:25:23.036 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:25:23.036 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:25:23.036 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:25:23.036 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:25:23.036 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:25:23.036 INFO [main] DataStoreTestBase.java:132 Test end: testSyncBigBlob 14:25:23.083 INFO [main] FileStore.java:466 TarMK closed: target\junit1350584405308229636\junit6739816403331306109 14:25:23.121 INFO [main] FileStore.java:466 TarMK closed: target\junit1350584405308229636\junit7862800850442952006