14:25:24.475 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1107085845286667683\junit5025071657868527490, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:25:24.475 INFO [main] FileStore.java:240 TarMK opened: target\junit1107085845286667683\junit5025071657868527490 (mmap=false) 14:25:24.490 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1107085845286667683\junit4041714097560529825, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:25:24.506 INFO [main] FileStore.java:240 TarMK opened: target\junit1107085845286667683\junit4041714097560529825 (mmap=false) 14:25:24.506 INFO [main] DataStoreTestBase.java:127 Test begin: testResilientSync 14:25:24.506 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:25:24.506 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:25:24.606 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x60182bf6]' 14:25:24.606 DEBUG [main] StandbyServer.java:249 Binding was successful 14:25:24.606 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 6422f5bb-dc45-4c1f-9804-cc09031599e1 for current head 14:25:24.606 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:62135 is allowed 14:25:24.606 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:25:24.606 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 6422f5bb-dc45-4c1f-9804-cc09031599e1 14:25:24.606 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 6422f5bb-dc45-4c1f-9804-cc09031599e1 14:25:28.611 ERROR [standby-run-44] StandbyClientSyncExecution.java:67 Unable to fetch remote head 14:25:28.611 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 6422f5bb-dc45-4c1f-9804-cc09031599e1 io.netty.handler.timeout.ReadTimeoutException: null 14:25:28.611 DEBUG [standby-run-44] StandbyClient.java:157 Channel closed 14:25:30.387 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 6cb7c8b8-88da-4667-a317-8334b2fedede.0000000c to client 6422f5bb-dc45-4c1f-9804-cc09031599e1 14:25:30.387 WARN [primary-1] ExceptionHandler.java:31 Exception caught on the server java.io.IOException: Eine bestehende Verbindung wurde softwaregesteuert durch den Hostcomputer abgebrochen at sun.nio.ch.SocketDispatcher.read0(Native Method) ~[na:1.8.0_151] at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43) ~[na:1.8.0_151] at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_151] at sun.nio.ch.IOUtil.read(IOUtil.java:192) ~[na:1.8.0_151] at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_151] at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288) ~[netty-buffer-4.1.14.Final.jar:4.1.14.Final] at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106) ~[netty-buffer-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:372) ~[netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123) ~[netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 14:25:30.829 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:25:30.829 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:25:30.829 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:25:30.830 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:25:30.832 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:25:30.850 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x43bba492]' 14:25:30.850 DEBUG [main] StandbyServer.java:249 Binding was successful 14:25:30.850 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client fe57d51d-7404-4ba2-aa0d-c32174cd5cef for current head 14:25:30.850 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:62183 is allowed 14:25:30.850 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:25:30.850 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client fe57d51d-7404-4ba2-aa0d-c32174cd5cef 14:25:30.850 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client fe57d51d-7404-4ba2-aa0d-c32174cd5cef 14:25:30.850 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 6cb7c8b8-88da-4667-a317-8334b2fedede.0000000c to client fe57d51d-7404-4ba2-aa0d-c32174cd5cef 14:25:30.850 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:25:30.850 DEBUG [standby-run-45] StandbyClientSyncExecution.java:103 Found missing segment 6cb7c8b8-88da-4667-a317-8334b2fedede 14:25:30.850 DEBUG [standby-run-45] StandbyClientSyncExecution.java:124 Inspecting segment 6cb7c8b8-88da-4667-a317-8334b2fedede 14:25:30.850 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client fe57d51d-7404-4ba2-aa0d-c32174cd5cef for references of segment 6cb7c8b8-88da-4667-a317-8334b2fedede 14:25:30.850 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:25:30.850 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 6cb7c8b8-88da-4667-a317-8334b2fedede for client fe57d51d-7404-4ba2-aa0d-c32174cd5cef 14:25:30.850 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 6cb7c8b8-88da-4667-a317-8334b2fedede to client fe57d51d-7404-4ba2-aa0d-c32174cd5cef 14:25:30.850 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:25:30.850 DEBUG [standby-run-45] StandbyClientSyncExecution.java:184 Found reference from 6cb7c8b8-88da-4667-a317-8334b2fedede to 84c2d3b7-540d-4112-a7a3-28602de1526a 14:25:30.850 DEBUG [standby-run-45] StandbyClientSyncExecution.java:124 Inspecting segment 84c2d3b7-540d-4112-a7a3-28602de1526a 14:25:30.850 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client fe57d51d-7404-4ba2-aa0d-c32174cd5cef for references of segment 84c2d3b7-540d-4112-a7a3-28602de1526a 14:25:30.850 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:25:30.850 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 84c2d3b7-540d-4112-a7a3-28602de1526a for client fe57d51d-7404-4ba2-aa0d-c32174cd5cef 14:25:30.850 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 84c2d3b7-540d-4112-a7a3-28602de1526a to client fe57d51d-7404-4ba2-aa0d-c32174cd5cef 14:25:30.850 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:25:30.865 INFO [standby-run-45] StandbyClientSyncExecution.java:196 Copying data segment 84c2d3b7-540d-4112-a7a3-28602de1526a from primary 14:25:30.865 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client fe57d51d-7404-4ba2-aa0d-c32174cd5cef for segment 84c2d3b7-540d-4112-a7a3-28602de1526a 14:25:30.865 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:25:30.865 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client fe57d51d-7404-4ba2-aa0d-c32174cd5cef 14:25:30.865 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 84c2d3b7-540d-4112-a7a3-28602de1526a for client fe57d51d-7404-4ba2-aa0d-c32174cd5cef 14:25:30.865 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client fe57d51d-7404-4ba2-aa0d-c32174cd5cef 14:25:30.865 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 84c2d3b7-540d-4112-a7a3-28602de1526a to client fe57d51d-7404-4ba2-aa0d-c32174cd5cef 14:25:30.865 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:25:30.865 INFO [standby-run-45] StandbyClientSyncExecution.java:196 Copying data segment 6cb7c8b8-88da-4667-a317-8334b2fedede from primary 14:25:30.865 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client fe57d51d-7404-4ba2-aa0d-c32174cd5cef for segment 6cb7c8b8-88da-4667-a317-8334b2fedede 14:25:30.865 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:25:30.865 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client fe57d51d-7404-4ba2-aa0d-c32174cd5cef 14:25:30.865 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 6cb7c8b8-88da-4667-a317-8334b2fedede for client fe57d51d-7404-4ba2-aa0d-c32174cd5cef 14:25:30.865 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client fe57d51d-7404-4ba2-aa0d-c32174cd5cef 14:25:30.865 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 6cb7c8b8-88da-4667-a317-8334b2fedede to client fe57d51d-7404-4ba2-aa0d-c32174cd5cef 14:25:30.865 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:25:31.333 INFO [standby-run-45] StandbyClientSyncExecution.java:82 updated head state successfully: true in 483ms. 14:25:31.334 DEBUG [standby-run-45] StandbyClient.java:157 Channel closed 14:25:33.555 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:25:33.555 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:25:33.555 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:25:33.571 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:25:33.618 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:25:33.640 INFO [main] DataStoreTestBase.java:132 Test end: testResilientSync 14:25:33.752 INFO [main] FileStore.java:466 TarMK closed: target\junit1107085845286667683\junit4041714097560529825 14:25:33.789 INFO [main] FileStore.java:466 TarMK closed: target\junit1107085845286667683\junit5025071657868527490 14:25:33.810 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit2915097913567592273\junit2512941806000356316, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:25:33.816 INFO [main] FileStore.java:240 TarMK opened: target\junit2915097913567592273\junit2512941806000356316 (mmap=false) 14:25:33.987 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit2915097913567592273\junit5608132171028081843, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:25:33.990 INFO [main] FileStore.java:240 TarMK opened: target\junit2915097913567592273\junit5608132171028081843 (mmap=false) 14:25:33.999 INFO [main] DataStoreTestBase.java:127 Test begin: testProxySkippedBytes 14:25:33.999 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:25:34.000 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:25:34.186 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x65b76bbe]' 14:25:34.188 DEBUG [main] StandbyServer.java:249 Binding was successful 14:25:34.269 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=-1, skip=100,1 14:25:34.273 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 62197 14:25:35.505 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 for current head 14:25:35.527 DEBUG [proxy-worker-153] ForwardHandler.java:88 Connected to remote host 14:25:35.528 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:62285 is allowed 14:25:35.529 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:25:35.529 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:35.529 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:35.530 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 8332dcd9-28f5-4dfc-a407-75b9d64ccea2.0000000c to client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:35.530 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:25:35.531 DEBUG [standby-run-46] StandbyClientSyncExecution.java:103 Found missing segment 8332dcd9-28f5-4dfc-a407-75b9d64ccea2 14:25:35.531 DEBUG [standby-run-46] StandbyClientSyncExecution.java:124 Inspecting segment 8332dcd9-28f5-4dfc-a407-75b9d64ccea2 14:25:35.532 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 for references of segment 8332dcd9-28f5-4dfc-a407-75b9d64ccea2 14:25:35.534 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:25:35.534 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 8332dcd9-28f5-4dfc-a407-75b9d64ccea2 for client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:35.535 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 8332dcd9-28f5-4dfc-a407-75b9d64ccea2 to client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:40.540 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 io.netty.handler.timeout.ReadTimeoutException: null 14:25:40.542 DEBUG [proxy-worker-153] ForwardHandler.java:98 Connection to remote host closed 14:25:40.549 DEBUG [standby-run-46] StandbyClient.java:157 Channel closed 14:25:40.550 ERROR [standby-run-46] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment 8332dcd9-28f5-4dfc-a407-75b9d64ccea2 from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:344) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxySkippedBytes(DataStoreTestBase.java:296) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 14:25:40.551 DEBUG [main] NetworkErrorProxy.java:116 Channel disconnected 14:25:40.551 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=-1, skip=-1,0 14:25:40.556 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 62197 14:25:40.557 DEBUG [proxy-worker-153] ForwardHandler.java:103 Group shut down 14:25:40.559 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 for current head 14:25:40.589 DEBUG [proxy-worker-154] ForwardHandler.java:88 Connected to remote host 14:25:40.590 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:62309 is allowed 14:25:40.590 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 14:25:40.590 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:40.591 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:40.591 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 8332dcd9-28f5-4dfc-a407-75b9d64ccea2.0000000c to client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:40.592 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 14:25:40.592 DEBUG [standby-run-47] StandbyClientSyncExecution.java:103 Found missing segment 8332dcd9-28f5-4dfc-a407-75b9d64ccea2 14:25:40.592 DEBUG [standby-run-47] StandbyClientSyncExecution.java:124 Inspecting segment 8332dcd9-28f5-4dfc-a407-75b9d64ccea2 14:25:40.593 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 for references of segment 8332dcd9-28f5-4dfc-a407-75b9d64ccea2 14:25:40.594 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:25:40.594 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 8332dcd9-28f5-4dfc-a407-75b9d64ccea2 for client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:40.594 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 8332dcd9-28f5-4dfc-a407-75b9d64ccea2 to client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:40.598 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:25:40.599 DEBUG [standby-run-47] StandbyClientSyncExecution.java:184 Found reference from 8332dcd9-28f5-4dfc-a407-75b9d64ccea2 to 8699ee4b-531f-4f7a-acfa-7a8d923240df 14:25:40.600 DEBUG [standby-run-47] StandbyClientSyncExecution.java:124 Inspecting segment 8699ee4b-531f-4f7a-acfa-7a8d923240df 14:25:40.600 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 for references of segment 8699ee4b-531f-4f7a-acfa-7a8d923240df 14:25:40.602 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:25:40.603 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 8699ee4b-531f-4f7a-acfa-7a8d923240df for client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:40.603 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 8699ee4b-531f-4f7a-acfa-7a8d923240df to client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:40.611 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:25:40.611 INFO [standby-run-47] StandbyClientSyncExecution.java:196 Copying data segment 8699ee4b-531f-4f7a-acfa-7a8d923240df from primary 14:25:40.612 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 for segment 8699ee4b-531f-4f7a-acfa-7a8d923240df 14:25:40.613 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:25:40.613 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:40.613 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 8699ee4b-531f-4f7a-acfa-7a8d923240df for client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:40.614 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:40.614 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 8699ee4b-531f-4f7a-acfa-7a8d923240df to client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:40.615 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:25:40.615 INFO [standby-run-47] StandbyClientSyncExecution.java:196 Copying data segment 8332dcd9-28f5-4dfc-a407-75b9d64ccea2 from primary 14:25:40.616 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 for segment 8332dcd9-28f5-4dfc-a407-75b9d64ccea2 14:25:40.616 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:25:40.617 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:40.617 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 8332dcd9-28f5-4dfc-a407-75b9d64ccea2 for client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:40.617 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:40.617 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 8332dcd9-28f5-4dfc-a407-75b9d64ccea2 to client ff392f3a-b2ba-4bb9-9b4a-a81c3b27eb24 14:25:40.618 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:25:41.301 INFO [standby-run-47] StandbyClientSyncExecution.java:82 updated head state successfully: true in 709ms. 14:25:41.302 DEBUG [standby-run-47] StandbyClient.java:157 Channel closed 14:25:41.302 DEBUG [proxy-worker-154] ForwardHandler.java:98 Connection to remote host closed 14:25:41.304 DEBUG [proxy-worker-154] ForwardHandler.java:103 Group shut down 14:25:43.610 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:25:43.611 DEBUG [main] NetworkErrorProxy.java:128 Channel closed 14:25:43.623 DEBUG [main] NetworkErrorProxy.java:134 Boss group shut down 14:25:43.629 DEBUG [main] NetworkErrorProxy.java:139 Worker group shut down 14:25:43.630 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:25:43.631 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:25:43.632 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:25:43.698 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:25:43.722 INFO [main] DataStoreTestBase.java:132 Test end: testProxySkippedBytes 14:25:43.854 INFO [main] FileStore.java:466 TarMK closed: target\junit2915097913567592273\junit5608132171028081843 14:25:43.926 INFO [main] FileStore.java:466 TarMK closed: target\junit2915097913567592273\junit2512941806000356316 14:25:43.950 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit5577292312640676390\junit5940586826244275397, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:25:43.954 INFO [main] FileStore.java:240 TarMK opened: target\junit5577292312640676390\junit5940586826244275397 (mmap=false) 14:25:43.973 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit5577292312640676390\junit6148263971284210009, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:25:43.976 INFO [main] FileStore.java:240 TarMK opened: target\junit5577292312640676390\junit6148263971284210009 (mmap=false) 14:25:44.031 INFO [main] DataStoreTestBase.java:127 Test begin: testSyncUpdatedBinaryProperty 14:25:44.031 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:25:44.031 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:25:44.070 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xfb16d34e]' 14:25:44.072 DEBUG [main] StandbyServer.java:249 Binding was successful 14:25:45.770 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e for current head 14:25:45.774 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:62352 is allowed 14:25:45.774 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:25:45.774 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:45.775 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:45.775 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 04d1235e-2878-466b-a14e-230da9dd840b.0000000c to client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:45.776 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:25:45.776 DEBUG [standby-run-48] StandbyClientSyncExecution.java:103 Found missing segment 04d1235e-2878-466b-a14e-230da9dd840b 14:25:45.776 DEBUG [standby-run-48] StandbyClientSyncExecution.java:124 Inspecting segment 04d1235e-2878-466b-a14e-230da9dd840b 14:25:45.777 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e for references of segment 04d1235e-2878-466b-a14e-230da9dd840b 14:25:45.777 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:25:45.778 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 04d1235e-2878-466b-a14e-230da9dd840b for client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:45.778 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 04d1235e-2878-466b-a14e-230da9dd840b to client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:45.779 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:25:45.782 DEBUG [standby-run-48] StandbyClientSyncExecution.java:184 Found reference from 04d1235e-2878-466b-a14e-230da9dd840b to 84bbba76-ea60-4781-a132-e97d9aa1ccd3 14:25:45.782 DEBUG [standby-run-48] StandbyClientSyncExecution.java:124 Inspecting segment 84bbba76-ea60-4781-a132-e97d9aa1ccd3 14:25:45.783 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e for references of segment 84bbba76-ea60-4781-a132-e97d9aa1ccd3 14:25:45.783 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:25:45.784 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 84bbba76-ea60-4781-a132-e97d9aa1ccd3 for client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:45.784 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 84bbba76-ea60-4781-a132-e97d9aa1ccd3 to client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:45.784 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:25:45.785 INFO [standby-run-48] StandbyClientSyncExecution.java:196 Copying data segment 84bbba76-ea60-4781-a132-e97d9aa1ccd3 from primary 14:25:45.785 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e for segment 84bbba76-ea60-4781-a132-e97d9aa1ccd3 14:25:45.786 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:25:45.786 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:45.786 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 84bbba76-ea60-4781-a132-e97d9aa1ccd3 for client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:45.786 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:45.787 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 84bbba76-ea60-4781-a132-e97d9aa1ccd3 to client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:45.787 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:25:45.787 INFO [standby-run-48] StandbyClientSyncExecution.java:196 Copying data segment 04d1235e-2878-466b-a14e-230da9dd840b from primary 14:25:45.788 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e for segment 04d1235e-2878-466b-a14e-230da9dd840b 14:25:45.788 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:25:45.789 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:45.789 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 04d1235e-2878-466b-a14e-230da9dd840b for client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:45.789 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:45.789 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 04d1235e-2878-466b-a14e-230da9dd840b to client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:45.790 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:25:46.890 INFO [standby-run-48] StandbyClientSyncExecution.java:82 updated head state successfully: true in 1114ms. 14:25:46.890 DEBUG [standby-run-48] StandbyClient.java:157 Channel closed 14:25:47.543 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e for current head 14:25:47.545 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:62355 is allowed 14:25:47.546 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 14:25:47.546 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:47.547 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:47.547 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head f7607e13-0929-4043-af77-3f16cabce886.00000006 to client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:47.548 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 14:25:47.549 DEBUG [standby-run-49] StandbyClientSyncExecution.java:103 Found missing segment f7607e13-0929-4043-af77-3f16cabce886 14:25:47.549 DEBUG [standby-run-49] StandbyClientSyncExecution.java:124 Inspecting segment f7607e13-0929-4043-af77-3f16cabce886 14:25:47.549 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e for references of segment f7607e13-0929-4043-af77-3f16cabce886 14:25:47.550 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:25:47.550 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment f7607e13-0929-4043-af77-3f16cabce886 for client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:47.550 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment f7607e13-0929-4043-af77-3f16cabce886 to client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:47.551 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:25:47.551 INFO [standby-run-49] StandbyClientSyncExecution.java:196 Copying data segment f7607e13-0929-4043-af77-3f16cabce886 from primary 14:25:47.552 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e for segment f7607e13-0929-4043-af77-3f16cabce886 14:25:47.552 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:25:47.552 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:47.552 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment f7607e13-0929-4043-af77-3f16cabce886 for client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:47.553 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 352 sent to client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:47.553 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment f7607e13-0929-4043-af77-3f16cabce886 to client e6aa7fc1-c964-4787-b2bd-2ee11a2a949e 14:25:47.553 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:25:47.935 INFO [standby-run-49] StandbyClientSyncExecution.java:82 updated head state successfully: true in 386ms. 14:25:47.937 DEBUG [standby-run-49] StandbyClient.java:157 Channel closed 14:25:50.240 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:25:50.240 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:25:50.241 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:25:50.242 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:25:50.249 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:25:50.249 INFO [main] DataStoreTestBase.java:132 Test end: testSyncUpdatedBinaryProperty 14:25:50.302 INFO [main] FileStore.java:466 TarMK closed: target\junit5577292312640676390\junit6148263971284210009 14:25:50.328 INFO [main] FileStore.java:466 TarMK closed: target\junit5577292312640676390\junit5940586826244275397 14:25:50.350 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit2971264525766394164\junit5728638970870084330, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:25:50.353 INFO [main] FileStore.java:240 TarMK opened: target\junit2971264525766394164\junit5728638970870084330 (mmap=false) 14:25:50.361 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit2971264525766394164\junit8820906339634513688, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:25:50.363 INFO [main] FileStore.java:240 TarMK opened: target\junit2971264525766394164\junit8820906339634513688 (mmap=false) 14:25:50.525 INFO [main] DataStoreTestBase.java:127 Test begin: testSync 14:25:50.525 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:25:50.525 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:25:50.688 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x6e348ef1]' 14:25:50.689 DEBUG [main] StandbyServer.java:249 Binding was successful 14:25:51.170 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 62b35638-2492-4048-a140-0367e69efb13 for current head 14:25:51.186 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:62401 is allowed 14:25:51.186 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:25:51.186 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 62b35638-2492-4048-a140-0367e69efb13 14:25:51.186 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 62b35638-2492-4048-a140-0367e69efb13 14:25:51.186 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head a2ace1cd-559d-445a-ad51-5be697138de0.0000000c to client 62b35638-2492-4048-a140-0367e69efb13 14:25:51.186 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:25:51.186 DEBUG [standby-run-50] StandbyClientSyncExecution.java:103 Found missing segment a2ace1cd-559d-445a-ad51-5be697138de0 14:25:51.186 DEBUG [standby-run-50] StandbyClientSyncExecution.java:124 Inspecting segment a2ace1cd-559d-445a-ad51-5be697138de0 14:25:51.201 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 62b35638-2492-4048-a140-0367e69efb13 for references of segment a2ace1cd-559d-445a-ad51-5be697138de0 14:25:51.201 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:25:51.201 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment a2ace1cd-559d-445a-ad51-5be697138de0 for client 62b35638-2492-4048-a140-0367e69efb13 14:25:51.201 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment a2ace1cd-559d-445a-ad51-5be697138de0 to client 62b35638-2492-4048-a140-0367e69efb13 14:25:51.201 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:25:51.201 DEBUG [standby-run-50] StandbyClientSyncExecution.java:184 Found reference from a2ace1cd-559d-445a-ad51-5be697138de0 to a20375c4-3fc9-4434-a08c-55cb925cf14f 14:25:51.201 DEBUG [standby-run-50] StandbyClientSyncExecution.java:124 Inspecting segment a20375c4-3fc9-4434-a08c-55cb925cf14f 14:25:51.201 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 62b35638-2492-4048-a140-0367e69efb13 for references of segment a20375c4-3fc9-4434-a08c-55cb925cf14f 14:25:51.201 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:25:51.201 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment a20375c4-3fc9-4434-a08c-55cb925cf14f for client 62b35638-2492-4048-a140-0367e69efb13 14:25:51.201 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment a20375c4-3fc9-4434-a08c-55cb925cf14f to client 62b35638-2492-4048-a140-0367e69efb13 14:25:51.201 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:25:51.201 INFO [standby-run-50] StandbyClientSyncExecution.java:196 Copying data segment a20375c4-3fc9-4434-a08c-55cb925cf14f from primary 14:25:51.201 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 62b35638-2492-4048-a140-0367e69efb13 for segment a20375c4-3fc9-4434-a08c-55cb925cf14f 14:25:51.201 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:25:51.201 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 62b35638-2492-4048-a140-0367e69efb13 14:25:51.201 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment a20375c4-3fc9-4434-a08c-55cb925cf14f for client 62b35638-2492-4048-a140-0367e69efb13 14:25:51.201 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client 62b35638-2492-4048-a140-0367e69efb13 14:25:51.201 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment a20375c4-3fc9-4434-a08c-55cb925cf14f to client 62b35638-2492-4048-a140-0367e69efb13 14:25:51.201 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:25:51.201 INFO [standby-run-50] StandbyClientSyncExecution.java:196 Copying data segment a2ace1cd-559d-445a-ad51-5be697138de0 from primary 14:25:51.201 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 62b35638-2492-4048-a140-0367e69efb13 for segment a2ace1cd-559d-445a-ad51-5be697138de0 14:25:51.201 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:25:51.201 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 62b35638-2492-4048-a140-0367e69efb13 14:25:51.201 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment a2ace1cd-559d-445a-ad51-5be697138de0 for client 62b35638-2492-4048-a140-0367e69efb13 14:25:51.201 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client 62b35638-2492-4048-a140-0367e69efb13 14:25:51.201 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment a2ace1cd-559d-445a-ad51-5be697138de0 to client 62b35638-2492-4048-a140-0367e69efb13 14:25:51.201 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:25:51.886 INFO [standby-run-50] StandbyClientSyncExecution.java:82 updated head state successfully: true in 700ms. 14:25:51.886 DEBUG [standby-run-50] StandbyClient.java:157 Channel closed 14:25:54.119 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:25:54.119 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:25:54.119 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:25:54.119 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:25:54.135 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:25:54.141 INFO [main] DataStoreTestBase.java:132 Test end: testSync 14:25:54.172 INFO [main] FileStore.java:466 TarMK closed: target\junit2971264525766394164\junit8820906339634513688 14:25:54.203 INFO [main] FileStore.java:466 TarMK closed: target\junit2971264525766394164\junit5728638970870084330 14:25:54.219 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1615510756392134073\junit5894102932877238925, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:25:54.219 INFO [main] FileStore.java:240 TarMK opened: target\junit1615510756392134073\junit5894102932877238925 (mmap=false) 14:25:54.241 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit1615510756392134073\junit1804996469860992114, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:25:54.241 INFO [main] FileStore.java:240 TarMK opened: target\junit1615510756392134073\junit1804996469860992114 (mmap=false) 14:25:54.256 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedIntermediateByteChange2 14:25:54.256 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:25:54.256 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:25:54.340 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x01f5a19b]' 14:25:54.341 DEBUG [main] StandbyServer.java:249 Binding was successful 14:25:54.357 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=150000, skip=0,0 14:25:54.357 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 62406 14:25:54.873 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 26f77437-290e-4f26-aab9-7749f3d532ff for current head 14:25:54.904 DEBUG [proxy-worker-161] ForwardHandler.java:88 Connected to remote host 14:25:54.904 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:62492 is allowed 14:25:54.904 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:25:54.904 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:54.904 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:54.904 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 38c0465b-b7c3-4d74-ad4a-54968666ddaf.0000000c to client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:54.904 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:25:54.904 DEBUG [standby-run-51] StandbyClientSyncExecution.java:103 Found missing segment 38c0465b-b7c3-4d74-ad4a-54968666ddaf 14:25:54.904 DEBUG [standby-run-51] StandbyClientSyncExecution.java:124 Inspecting segment 38c0465b-b7c3-4d74-ad4a-54968666ddaf 14:25:54.904 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 26f77437-290e-4f26-aab9-7749f3d532ff for references of segment 38c0465b-b7c3-4d74-ad4a-54968666ddaf 14:25:54.904 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:25:54.904 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 38c0465b-b7c3-4d74-ad4a-54968666ddaf for client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:54.904 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 38c0465b-b7c3-4d74-ad4a-54968666ddaf to client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:54.904 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:25:54.904 DEBUG [standby-run-51] StandbyClientSyncExecution.java:184 Found reference from 38c0465b-b7c3-4d74-ad4a-54968666ddaf to c7c228d3-52cd-47b4-a716-6ceff61b0aa5 14:25:54.904 DEBUG [standby-run-51] StandbyClientSyncExecution.java:124 Inspecting segment c7c228d3-52cd-47b4-a716-6ceff61b0aa5 14:25:54.904 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 26f77437-290e-4f26-aab9-7749f3d532ff for references of segment c7c228d3-52cd-47b4-a716-6ceff61b0aa5 14:25:54.904 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:25:54.904 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment c7c228d3-52cd-47b4-a716-6ceff61b0aa5 for client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:54.904 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment c7c228d3-52cd-47b4-a716-6ceff61b0aa5 to client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:54.904 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:25:54.904 INFO [standby-run-51] StandbyClientSyncExecution.java:196 Copying data segment c7c228d3-52cd-47b4-a716-6ceff61b0aa5 from primary 14:25:54.904 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 26f77437-290e-4f26-aab9-7749f3d532ff for segment c7c228d3-52cd-47b4-a716-6ceff61b0aa5 14:25:54.904 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:25:54.904 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:54.904 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment c7c228d3-52cd-47b4-a716-6ceff61b0aa5 for client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:54.919 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:54.919 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment c7c228d3-52cd-47b4-a716-6ceff61b0aa5 to client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:54.919 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:25:54.919 INFO [standby-run-51] StandbyClientSyncExecution.java:196 Copying data segment 38c0465b-b7c3-4d74-ad4a-54968666ddaf from primary 14:25:54.919 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 26f77437-290e-4f26-aab9-7749f3d532ff for segment 38c0465b-b7c3-4d74-ad4a-54968666ddaf 14:25:54.919 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:25:54.919 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:54.919 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 38c0465b-b7c3-4d74-ad4a-54968666ddaf for client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:54.919 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:54.919 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 38c0465b-b7c3-4d74-ad4a-54968666ddaf to client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:54.919 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:25:55.675 INFO [standby-run-51] StandbyClientSyncExecution.java:82 updated head state successfully: true in 771ms. 14:25:55.675 DEBUG [standby-run-51] StandbyClient.java:157 Channel closed 14:25:55.675 DEBUG [proxy-worker-161] ForwardHandler.java:98 Connection to remote host closed 14:25:55.675 DEBUG [main] NetworkErrorProxy.java:116 Channel disconnected 14:25:55.675 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=-1, skip=-1,0 14:25:55.675 DEBUG [proxy-worker-161] ForwardHandler.java:103 Group shut down 14:25:55.675 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 62406 14:25:56.143 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client 26f77437-290e-4f26-aab9-7749f3d532ff for current head 14:25:56.174 DEBUG [proxy-worker-162] ForwardHandler.java:88 Connected to remote host 14:25:56.174 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:62512 is allowed 14:25:56.174 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 14:25:56.174 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:56.174 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:56.174 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 1180362d-ab61-4e10-a83a-cc484d3e6c67.00000006 to client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:56.190 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 14:25:56.190 DEBUG [standby-run-52] StandbyClientSyncExecution.java:103 Found missing segment 1180362d-ab61-4e10-a83a-cc484d3e6c67 14:25:56.190 DEBUG [standby-run-52] StandbyClientSyncExecution.java:124 Inspecting segment 1180362d-ab61-4e10-a83a-cc484d3e6c67 14:25:56.190 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 26f77437-290e-4f26-aab9-7749f3d532ff for references of segment 1180362d-ab61-4e10-a83a-cc484d3e6c67 14:25:56.190 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:25:56.190 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 1180362d-ab61-4e10-a83a-cc484d3e6c67 for client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:56.190 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 1180362d-ab61-4e10-a83a-cc484d3e6c67 to client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:56.190 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:25:56.190 INFO [standby-run-52] StandbyClientSyncExecution.java:196 Copying data segment 1180362d-ab61-4e10-a83a-cc484d3e6c67 from primary 14:25:56.190 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 26f77437-290e-4f26-aab9-7749f3d532ff for segment 1180362d-ab61-4e10-a83a-cc484d3e6c67 14:25:56.190 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:25:56.190 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:56.190 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 1180362d-ab61-4e10-a83a-cc484d3e6c67 for client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:56.190 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 352 sent to client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:56.190 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 1180362d-ab61-4e10-a83a-cc484d3e6c67 to client 26f77437-290e-4f26-aab9-7749f3d532ff 14:25:56.190 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:25:56.590 INFO [standby-run-52] StandbyClientSyncExecution.java:82 updated head state successfully: true in 400ms. 14:25:56.590 DEBUG [standby-run-52] StandbyClient.java:157 Channel closed 14:25:56.590 DEBUG [proxy-worker-162] ForwardHandler.java:98 Connection to remote host closed 14:25:56.606 DEBUG [proxy-worker-162] ForwardHandler.java:103 Group shut down 14:25:58.937 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:25:58.937 DEBUG [main] NetworkErrorProxy.java:128 Channel closed 14:25:58.940 DEBUG [main] NetworkErrorProxy.java:134 Boss group shut down 14:25:58.943 DEBUG [main] NetworkErrorProxy.java:139 Worker group shut down 14:25:58.943 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:25:58.943 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:25:58.943 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:25:58.943 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:25:58.943 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedIntermediateByteChange2 14:25:58.975 INFO [main] FileStore.java:466 TarMK closed: target\junit1615510756392134073\junit1804996469860992114 14:25:59.022 INFO [main] FileStore.java:466 TarMK closed: target\junit1615510756392134073\junit5894102932877238925 14:25:59.041 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit29420783196377753\junit9184933657300260795, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:25:59.044 INFO [main] FileStore.java:240 TarMK opened: target\junit29420783196377753\junit9184933657300260795 (mmap=false) 14:25:59.175 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit29420783196377753\junit3440729913992887090, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:25:59.190 INFO [main] FileStore.java:240 TarMK opened: target\junit29420783196377753\junit3440729913992887090 (mmap=false) 14:25:59.190 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedIntermediateByteChange 14:25:59.190 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:25:59.190 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:25:59.275 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xef5a6f9d]' 14:25:59.275 DEBUG [main] StandbyServer.java:249 Binding was successful 14:25:59.306 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=150, skip=0,0 14:25:59.306 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 62516 14:25:59.691 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 for current head 14:25:59.722 DEBUG [proxy-worker-169] ForwardHandler.java:88 Connected to remote host 14:25:59.722 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:62603 is allowed 14:25:59.722 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:25:59.722 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:25:59.722 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:25:59.722 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 80ac3ea3-7606-45c9-aa7e-10370abb1ca5.0000000c to client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:25:59.722 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:25:59.722 DEBUG [standby-run-53] StandbyClientSyncExecution.java:103 Found missing segment 80ac3ea3-7606-45c9-aa7e-10370abb1ca5 14:25:59.722 DEBUG [standby-run-53] StandbyClientSyncExecution.java:124 Inspecting segment 80ac3ea3-7606-45c9-aa7e-10370abb1ca5 14:25:59.722 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 for references of segment 80ac3ea3-7606-45c9-aa7e-10370abb1ca5 14:25:59.722 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:25:59.722 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 80ac3ea3-7606-45c9-aa7e-10370abb1ca5 for client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:25:59.722 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 80ac3ea3-7606-45c9-aa7e-10370abb1ca5 to client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:25:59.722 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 io.netty.handler.codec.compression.DecompressionException: mismatching checksum: a28269ab (expected: 459b3c70) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 14:26:04.745 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 io.netty.handler.timeout.ReadTimeoutException: null 14:26:04.746 DEBUG [standby-run-53] StandbyClient.java:157 Channel closed 14:26:04.747 DEBUG [proxy-worker-169] ForwardHandler.java:98 Connection to remote host closed 14:26:04.748 ERROR [standby-run-53] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment 80ac3ea3-7606-45c9-aa7e-10370abb1ca5 from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:344) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByteChange(DataStoreTestBase.java:321) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 14:26:04.751 DEBUG [main] NetworkErrorProxy.java:116 Channel disconnected 14:26:04.751 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=-1, skip=-1,0 14:26:04.751 DEBUG [proxy-worker-169] ForwardHandler.java:103 Group shut down 14:26:04.751 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 62516 14:26:05.182 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 for current head 14:26:05.197 DEBUG [proxy-worker-170] ForwardHandler.java:88 Connected to remote host 14:26:05.197 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:62628 is allowed 14:26:05.197 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 14:26:05.197 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.197 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.197 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head bb9bf894-bb32-4c9f-a4c3-090c8bbc8189.00000006 to client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.197 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 14:26:05.197 DEBUG [standby-run-54] StandbyClientSyncExecution.java:103 Found missing segment bb9bf894-bb32-4c9f-a4c3-090c8bbc8189 14:26:05.197 DEBUG [standby-run-54] StandbyClientSyncExecution.java:124 Inspecting segment bb9bf894-bb32-4c9f-a4c3-090c8bbc8189 14:26:05.197 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 for references of segment bb9bf894-bb32-4c9f-a4c3-090c8bbc8189 14:26:05.197 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:26:05.197 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment bb9bf894-bb32-4c9f-a4c3-090c8bbc8189 for client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.197 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment bb9bf894-bb32-4c9f-a4c3-090c8bbc8189 to client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.197 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:26:05.197 DEBUG [standby-run-54] StandbyClientSyncExecution.java:184 Found reference from bb9bf894-bb32-4c9f-a4c3-090c8bbc8189 to 80ac3ea3-7606-45c9-aa7e-10370abb1ca5 14:26:05.197 DEBUG [standby-run-54] StandbyClientSyncExecution.java:184 Found reference from bb9bf894-bb32-4c9f-a4c3-090c8bbc8189 to 5e948a91-3429-41e3-a110-7f9ed2fb79b7 14:26:05.197 DEBUG [standby-run-54] StandbyClientSyncExecution.java:124 Inspecting segment 80ac3ea3-7606-45c9-aa7e-10370abb1ca5 14:26:05.197 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 for references of segment 80ac3ea3-7606-45c9-aa7e-10370abb1ca5 14:26:05.197 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:26:05.197 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 80ac3ea3-7606-45c9-aa7e-10370abb1ca5 for client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.197 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 80ac3ea3-7606-45c9-aa7e-10370abb1ca5 to client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.197 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:26:05.197 DEBUG [standby-run-54] StandbyClientSyncExecution.java:124 Inspecting segment 5e948a91-3429-41e3-a110-7f9ed2fb79b7 14:26:05.197 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 for references of segment 5e948a91-3429-41e3-a110-7f9ed2fb79b7 14:26:05.197 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:26:05.197 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 5e948a91-3429-41e3-a110-7f9ed2fb79b7 for client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.197 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 5e948a91-3429-41e3-a110-7f9ed2fb79b7 to client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.213 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:26:05.213 INFO [standby-run-54] StandbyClientSyncExecution.java:196 Copying data segment 5e948a91-3429-41e3-a110-7f9ed2fb79b7 from primary 14:26:05.213 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 for segment 5e948a91-3429-41e3-a110-7f9ed2fb79b7 14:26:05.213 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:26:05.213 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.213 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 5e948a91-3429-41e3-a110-7f9ed2fb79b7 for client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.213 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.213 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 5e948a91-3429-41e3-a110-7f9ed2fb79b7 to client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.213 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:26:05.213 INFO [standby-run-54] StandbyClientSyncExecution.java:196 Copying data segment 80ac3ea3-7606-45c9-aa7e-10370abb1ca5 from primary 14:26:05.213 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 for segment 80ac3ea3-7606-45c9-aa7e-10370abb1ca5 14:26:05.213 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:26:05.213 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.213 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 80ac3ea3-7606-45c9-aa7e-10370abb1ca5 for client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.213 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.213 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 80ac3ea3-7606-45c9-aa7e-10370abb1ca5 to client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.213 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:26:05.213 INFO [standby-run-54] StandbyClientSyncExecution.java:196 Copying data segment bb9bf894-bb32-4c9f-a4c3-090c8bbc8189 from primary 14:26:05.213 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 for segment bb9bf894-bb32-4c9f-a4c3-090c8bbc8189 14:26:05.213 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:26:05.213 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.213 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment bb9bf894-bb32-4c9f-a4c3-090c8bbc8189 for client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.213 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 352 sent to client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.213 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment bb9bf894-bb32-4c9f-a4c3-090c8bbc8189 to client 2bde9f5a-1bd7-40a2-8015-edd5151fe2d1 14:26:05.213 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:26:05.547 INFO [standby-run-54] StandbyClientSyncExecution.java:82 updated head state successfully: true in 350ms. 14:26:05.548 DEBUG [standby-run-54] StandbyClient.java:157 Channel closed 14:26:05.548 DEBUG [proxy-worker-170] ForwardHandler.java:98 Connection to remote host closed 14:26:05.552 DEBUG [proxy-worker-170] ForwardHandler.java:103 Group shut down 14:26:07.901 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:26:07.901 DEBUG [main] NetworkErrorProxy.java:128 Channel closed 14:26:07.901 DEBUG [main] NetworkErrorProxy.java:134 Boss group shut down 14:26:07.901 DEBUG [main] NetworkErrorProxy.java:139 Worker group shut down 14:26:07.901 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:26:07.901 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:26:07.901 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:26:07.901 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:26:07.901 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedIntermediateByteChange 14:26:07.932 INFO [main] FileStore.java:466 TarMK closed: target\junit29420783196377753\junit3440729913992887090 14:26:07.986 INFO [main] FileStore.java:466 TarMK closed: target\junit29420783196377753\junit9184933657300260795 14:26:08.001 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit2279813029825457242\junit1853095065108597051, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:26:08.001 INFO [main] FileStore.java:240 TarMK opened: target\junit2279813029825457242\junit1853095065108597051 (mmap=false) 14:26:10.235 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit2279813029825457242\junit5731013382415393608, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:26:10.258 INFO [main] FileStore.java:240 TarMK opened: target\junit2279813029825457242\junit5731013382415393608 (mmap=false) 14:26:10.258 INFO [main] DataStoreTestBase.java:127 Test begin: testProxySkippedBytesIntermediateChange 14:26:10.258 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:26:10.258 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:26:10.389 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x032bc403]' 14:26:10.389 DEBUG [main] StandbyServer.java:249 Binding was successful 14:26:10.420 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=-1, skip=100,1 14:26:10.420 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 62634 14:26:10.858 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 506e8854-d363-46aa-9eb1-1fdaa223c02d for current head 14:26:10.905 DEBUG [proxy-worker-177] ForwardHandler.java:88 Connected to remote host 14:26:10.905 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:62722 is allowed 14:26:10.905 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:26:10.905 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:10.905 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:10.905 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head eafaf346-4952-4252-abf5-fc85dba81508.0000000c to client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:10.905 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:26:10.905 DEBUG [standby-run-55] StandbyClientSyncExecution.java:103 Found missing segment eafaf346-4952-4252-abf5-fc85dba81508 14:26:10.905 DEBUG [standby-run-55] StandbyClientSyncExecution.java:124 Inspecting segment eafaf346-4952-4252-abf5-fc85dba81508 14:26:10.905 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 506e8854-d363-46aa-9eb1-1fdaa223c02d for references of segment eafaf346-4952-4252-abf5-fc85dba81508 14:26:10.920 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:26:10.920 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment eafaf346-4952-4252-abf5-fc85dba81508 for client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:10.920 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment eafaf346-4952-4252-abf5-fc85dba81508 to client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:15.923 DEBUG [standby-run-55] StandbyClient.java:157 Channel closed 14:26:15.923 DEBUG [proxy-worker-177] ForwardHandler.java:98 Connection to remote host closed 14:26:15.923 ERROR [standby-run-55] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment eafaf346-4952-4252-abf5-fc85dba81508 from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:344) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxySkippedBytesIntermediateChange(DataStoreTestBase.java:301) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 14:26:15.923 DEBUG [main] NetworkErrorProxy.java:116 Channel disconnected 14:26:15.923 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=-1, skip=-1,0 14:26:15.923 DEBUG [proxy-worker-177] ForwardHandler.java:103 Group shut down 14:26:15.923 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 62634 14:26:16.461 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client 506e8854-d363-46aa-9eb1-1fdaa223c02d for current head 14:26:16.477 DEBUG [proxy-worker-178] ForwardHandler.java:88 Connected to remote host 14:26:16.477 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:62748 is allowed 14:26:16.477 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 14:26:16.477 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.477 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.477 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head f8186934-3a69-49ee-af6f-3034401e3073.00000006 to client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.477 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 14:26:16.477 DEBUG [standby-run-56] StandbyClientSyncExecution.java:103 Found missing segment f8186934-3a69-49ee-af6f-3034401e3073 14:26:16.477 DEBUG [standby-run-56] StandbyClientSyncExecution.java:124 Inspecting segment f8186934-3a69-49ee-af6f-3034401e3073 14:26:16.477 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 506e8854-d363-46aa-9eb1-1fdaa223c02d for references of segment f8186934-3a69-49ee-af6f-3034401e3073 14:26:16.477 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:26:16.477 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment f8186934-3a69-49ee-af6f-3034401e3073 for client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.477 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment f8186934-3a69-49ee-af6f-3034401e3073 to client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.477 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:26:16.477 DEBUG [standby-run-56] StandbyClientSyncExecution.java:184 Found reference from f8186934-3a69-49ee-af6f-3034401e3073 to eafaf346-4952-4252-abf5-fc85dba81508 14:26:16.477 DEBUG [standby-run-56] StandbyClientSyncExecution.java:184 Found reference from f8186934-3a69-49ee-af6f-3034401e3073 to 25fd89fa-b2a9-4ea3-a30b-852b4657c44f 14:26:16.477 DEBUG [standby-run-56] StandbyClientSyncExecution.java:124 Inspecting segment eafaf346-4952-4252-abf5-fc85dba81508 14:26:16.492 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 506e8854-d363-46aa-9eb1-1fdaa223c02d for references of segment eafaf346-4952-4252-abf5-fc85dba81508 14:26:16.492 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:26:16.492 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment eafaf346-4952-4252-abf5-fc85dba81508 for client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.492 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment eafaf346-4952-4252-abf5-fc85dba81508 to client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.492 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:26:16.492 DEBUG [standby-run-56] StandbyClientSyncExecution.java:124 Inspecting segment 25fd89fa-b2a9-4ea3-a30b-852b4657c44f 14:26:16.492 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 506e8854-d363-46aa-9eb1-1fdaa223c02d for references of segment 25fd89fa-b2a9-4ea3-a30b-852b4657c44f 14:26:16.492 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:26:16.492 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 25fd89fa-b2a9-4ea3-a30b-852b4657c44f for client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.492 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 25fd89fa-b2a9-4ea3-a30b-852b4657c44f to client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.492 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:26:16.492 INFO [standby-run-56] StandbyClientSyncExecution.java:196 Copying data segment 25fd89fa-b2a9-4ea3-a30b-852b4657c44f from primary 14:26:16.492 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 506e8854-d363-46aa-9eb1-1fdaa223c02d for segment 25fd89fa-b2a9-4ea3-a30b-852b4657c44f 14:26:16.492 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:26:16.492 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.492 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 25fd89fa-b2a9-4ea3-a30b-852b4657c44f for client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.492 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.492 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 25fd89fa-b2a9-4ea3-a30b-852b4657c44f to client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.492 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:26:16.492 INFO [standby-run-56] StandbyClientSyncExecution.java:196 Copying data segment eafaf346-4952-4252-abf5-fc85dba81508 from primary 14:26:16.492 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 506e8854-d363-46aa-9eb1-1fdaa223c02d for segment eafaf346-4952-4252-abf5-fc85dba81508 14:26:16.492 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:26:16.492 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.492 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment eafaf346-4952-4252-abf5-fc85dba81508 for client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.492 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.492 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment eafaf346-4952-4252-abf5-fc85dba81508 to client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.492 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:26:16.492 INFO [standby-run-56] StandbyClientSyncExecution.java:196 Copying data segment f8186934-3a69-49ee-af6f-3034401e3073 from primary 14:26:16.492 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 506e8854-d363-46aa-9eb1-1fdaa223c02d for segment f8186934-3a69-49ee-af6f-3034401e3073 14:26:16.492 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:26:16.492 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.492 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment f8186934-3a69-49ee-af6f-3034401e3073 for client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.492 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 352 sent to client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.492 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment f8186934-3a69-49ee-af6f-3034401e3073 to client 506e8854-d363-46aa-9eb1-1fdaa223c02d 14:26:16.492 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:26:16.808 INFO [standby-run-56] StandbyClientSyncExecution.java:82 updated head state successfully: true in 331ms. 14:26:16.808 DEBUG [standby-run-56] StandbyClient.java:157 Channel closed 14:26:16.808 DEBUG [proxy-worker-178] ForwardHandler.java:98 Connection to remote host closed 14:26:16.823 DEBUG [proxy-worker-178] ForwardHandler.java:103 Group shut down 14:26:19.157 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:26:19.158 DEBUG [main] NetworkErrorProxy.java:128 Channel closed 14:26:19.162 DEBUG [main] NetworkErrorProxy.java:134 Boss group shut down 14:26:19.162 DEBUG [main] NetworkErrorProxy.java:139 Worker group shut down 14:26:19.162 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:26:19.162 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:26:19.162 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:26:19.178 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:26:19.178 INFO [main] DataStoreTestBase.java:132 Test end: testProxySkippedBytesIntermediateChange 14:26:19.240 INFO [main] FileStore.java:466 TarMK closed: target\junit2279813029825457242\junit5731013382415393608 14:26:19.262 INFO [main] FileStore.java:466 TarMK closed: target\junit2279813029825457242\junit1853095065108597051 14:26:19.278 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit4758784154598215993\junit5194170974441362376, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:26:19.278 INFO [main] FileStore.java:240 TarMK opened: target\junit4758784154598215993\junit5194170974441362376 (mmap=false) 14:26:19.593 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit4758784154598215993\junit5555674804826095410, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:26:19.609 INFO [main] FileStore.java:240 TarMK opened: target\junit4758784154598215993\junit5555674804826095410 (mmap=false) 14:26:19.724 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedIntermediateByte 14:26:19.724 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:26:19.724 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:26:19.825 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xea39eccb]' 14:26:19.825 DEBUG [main] StandbyServer.java:249 Binding was successful 14:26:19.841 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=150, skip=0,0 14:26:19.841 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 62754 14:26:20.226 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 for current head 14:26:20.241 DEBUG [proxy-worker-185] ForwardHandler.java:88 Connected to remote host 14:26:20.241 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:62839 is allowed 14:26:20.241 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:26:20.241 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:20.241 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:20.241 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 694560e4-e4f7-4c19-a120-166a97eca9a9.0000000c to client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:20.257 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:26:20.258 DEBUG [standby-run-57] StandbyClientSyncExecution.java:103 Found missing segment 694560e4-e4f7-4c19-a120-166a97eca9a9 14:26:20.258 DEBUG [standby-run-57] StandbyClientSyncExecution.java:124 Inspecting segment 694560e4-e4f7-4c19-a120-166a97eca9a9 14:26:20.258 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 for references of segment 694560e4-e4f7-4c19-a120-166a97eca9a9 14:26:20.259 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:26:20.259 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 694560e4-e4f7-4c19-a120-166a97eca9a9 for client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:20.260 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 694560e4-e4f7-4c19-a120-166a97eca9a9 to client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:20.261 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 io.netty.handler.codec.compression.DecompressionException: mismatching checksum: 5c275f0b (expected: a2824650) at io.netty.handler.codec.compression.Snappy.validateChecksum(Snappy.java:640) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:181) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 14:26:25.266 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 io.netty.handler.timeout.ReadTimeoutException: null 14:26:25.267 DEBUG [standby-run-57] StandbyClient.java:157 Channel closed 14:26:25.267 DEBUG [proxy-worker-185] ForwardHandler.java:98 Connection to remote host closed 14:26:25.268 ERROR [standby-run-57] StandbyClientSync.java:171 Failed synchronizing state. java.lang.IllegalStateException: Unable to read references of segment 694560e4-e4f7-4c19-a120-166a97eca9a9 from primary at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.readReferences(StandbyClientSyncExecution.java:206) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.copySegmentHierarchyFromPrimary(StandbyClientSyncExecution.java:140) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.compareAgainstBaseState(StandbyClientSyncExecution.java:104) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSyncExecution.execute(StandbyClientSyncExecution.java:79) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.client.StandbyClientSync.run(StandbyClientSync.java:157) ~[oak-segment-tar-1.8-SNAPSHOT.jar:1.8-SNAPSHOT] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.useProxy(DataStoreTestBase.java:344) [test-classes/:na] at org.apache.jackrabbit.oak.segment.standby.DataStoreTestBase.testProxyFlippedIntermediateByte(DataStoreTestBase.java:311) [test-classes/:na] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_151] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_151] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_151] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_151] at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) [junit-4.12.jar:4.12] at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) [junit-4.12.jar:4.12] at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) [junit-4.12.jar:4.12] at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) [junit-4.12.jar:4.12] at org.junit.rules.RunRules.evaluate(RunRules.java:20) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) [junit-4.12.jar:4.12] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) [junit-4.12.jar:4.12] at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) [junit-4.12.jar:4.12] at org.junit.runners.ParentRunner.run(ParentRunner.java:363) [junit-4.12.jar:4.12] at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160) [surefire-junit4-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:373) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:334) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:119) [surefire-booter-2.20.1.jar:2.20.1] at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:407) [surefire-booter-2.20.1.jar:2.20.1] 14:26:25.269 DEBUG [main] NetworkErrorProxy.java:116 Channel disconnected 14:26:25.269 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=-1, skip=-1,0 14:26:25.272 DEBUG [proxy-worker-185] ForwardHandler.java:103 Group shut down 14:26:25.272 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 62754 14:26:25.273 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 for current head 14:26:25.288 DEBUG [proxy-worker-186] ForwardHandler.java:88 Connected to remote host 14:26:25.288 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:62867 is allowed 14:26:25.288 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 14:26:25.288 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:25.288 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:25.288 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 694560e4-e4f7-4c19-a120-166a97eca9a9.0000000c to client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:25.288 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 14:26:25.288 DEBUG [standby-run-58] StandbyClientSyncExecution.java:103 Found missing segment 694560e4-e4f7-4c19-a120-166a97eca9a9 14:26:25.288 DEBUG [standby-run-58] StandbyClientSyncExecution.java:124 Inspecting segment 694560e4-e4f7-4c19-a120-166a97eca9a9 14:26:25.288 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 for references of segment 694560e4-e4f7-4c19-a120-166a97eca9a9 14:26:25.288 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:26:25.288 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 694560e4-e4f7-4c19-a120-166a97eca9a9 for client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:25.288 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 694560e4-e4f7-4c19-a120-166a97eca9a9 to client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:25.288 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:26:25.288 DEBUG [standby-run-58] StandbyClientSyncExecution.java:184 Found reference from 694560e4-e4f7-4c19-a120-166a97eca9a9 to 5f020640-f80c-4b73-aa16-d6b2ee649eab 14:26:25.288 DEBUG [standby-run-58] StandbyClientSyncExecution.java:124 Inspecting segment 5f020640-f80c-4b73-aa16-d6b2ee649eab 14:26:25.288 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 for references of segment 5f020640-f80c-4b73-aa16-d6b2ee649eab 14:26:25.288 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:26:25.288 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 5f020640-f80c-4b73-aa16-d6b2ee649eab for client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:25.288 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 5f020640-f80c-4b73-aa16-d6b2ee649eab to client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:25.288 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:26:25.288 INFO [standby-run-58] StandbyClientSyncExecution.java:196 Copying data segment 5f020640-f80c-4b73-aa16-d6b2ee649eab from primary 14:26:25.288 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 for segment 5f020640-f80c-4b73-aa16-d6b2ee649eab 14:26:25.288 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:26:25.288 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:25.288 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 5f020640-f80c-4b73-aa16-d6b2ee649eab for client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:25.288 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:25.288 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 5f020640-f80c-4b73-aa16-d6b2ee649eab to client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:25.288 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:26:25.288 INFO [standby-run-58] StandbyClientSyncExecution.java:196 Copying data segment 694560e4-e4f7-4c19-a120-166a97eca9a9 from primary 14:26:25.288 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 for segment 694560e4-e4f7-4c19-a120-166a97eca9a9 14:26:25.288 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:26:25.288 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:25.288 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 694560e4-e4f7-4c19-a120-166a97eca9a9 for client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:25.288 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:25.288 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 694560e4-e4f7-4c19-a120-166a97eca9a9 to client a6875d62-2db5-4e7a-a6f2-5a57d94aae79 14:26:25.288 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:26:25.304 INFO [standby-run-58] StandbyClientSyncExecution.java:82 updated head state successfully: true in 16ms. 14:26:25.304 DEBUG [standby-run-58] StandbyClient.java:157 Channel closed 14:26:25.304 DEBUG [proxy-worker-186] ForwardHandler.java:98 Connection to remote host closed 14:26:25.304 DEBUG [proxy-worker-186] ForwardHandler.java:103 Group shut down 14:26:27.608 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:26:27.608 DEBUG [main] NetworkErrorProxy.java:128 Channel closed 14:26:27.610 DEBUG [main] NetworkErrorProxy.java:134 Boss group shut down 14:26:27.614 DEBUG [main] NetworkErrorProxy.java:139 Worker group shut down 14:26:27.614 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:26:27.615 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:26:27.615 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:26:27.619 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:26:27.627 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedIntermediateByte 14:26:28.376 INFO [main] FileStore.java:466 TarMK closed: target\junit4758784154598215993\junit5555674804826095410 14:26:28.415 INFO [main] FileStore.java:466 TarMK closed: target\junit4758784154598215993\junit5194170974441362376 14:26:28.440 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit32718061797662336\junit7700110466855248011, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:26:28.443 INFO [main] FileStore.java:240 TarMK opened: target\junit32718061797662336\junit7700110466855248011 (mmap=false) 14:26:28.450 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit32718061797662336\junit6944884391267138663, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:26:28.453 INFO [main] FileStore.java:240 TarMK opened: target\junit32718061797662336\junit6944884391267138663 (mmap=false) 14:26:28.454 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedIntermediateByte2 14:26:28.454 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:26:28.454 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:26:28.565 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xd2b88792]' 14:26:28.566 DEBUG [main] StandbyServer.java:249 Binding was successful 14:26:28.598 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=150000, skip=0,0 14:26:28.599 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 62875 14:26:29.140 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 for current head 14:26:29.161 DEBUG [proxy-worker-193] ForwardHandler.java:88 Connected to remote host 14:26:29.162 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:62960 is allowed 14:26:29.163 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:26:29.163 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 14:26:29.164 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 14:26:29.165 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head 020e218c-9ae2-46af-a581-0272a3dd6fee.0000000c to client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 14:26:29.165 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:26:29.166 DEBUG [standby-run-59] StandbyClientSyncExecution.java:103 Found missing segment 020e218c-9ae2-46af-a581-0272a3dd6fee 14:26:29.166 DEBUG [standby-run-59] StandbyClientSyncExecution.java:124 Inspecting segment 020e218c-9ae2-46af-a581-0272a3dd6fee 14:26:29.166 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 for references of segment 020e218c-9ae2-46af-a581-0272a3dd6fee 14:26:29.167 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:26:29.167 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment 020e218c-9ae2-46af-a581-0272a3dd6fee for client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 14:26:29.167 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment 020e218c-9ae2-46af-a581-0272a3dd6fee to client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 14:26:29.168 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:26:29.168 DEBUG [standby-run-59] StandbyClientSyncExecution.java:184 Found reference from 020e218c-9ae2-46af-a581-0272a3dd6fee to fa229a47-8269-405f-a4eb-241278a933b2 14:26:29.168 DEBUG [standby-run-59] StandbyClientSyncExecution.java:124 Inspecting segment fa229a47-8269-405f-a4eb-241278a933b2 14:26:29.168 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 for references of segment fa229a47-8269-405f-a4eb-241278a933b2 14:26:29.169 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:26:29.169 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment fa229a47-8269-405f-a4eb-241278a933b2 for client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 14:26:29.169 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment fa229a47-8269-405f-a4eb-241278a933b2 to client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 14:26:29.169 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:26:29.170 INFO [standby-run-59] StandbyClientSyncExecution.java:196 Copying data segment fa229a47-8269-405f-a4eb-241278a933b2 from primary 14:26:29.170 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 for segment fa229a47-8269-405f-a4eb-241278a933b2 14:26:29.170 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:26:29.170 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 14:26:29.170 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment fa229a47-8269-405f-a4eb-241278a933b2 for client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 14:26:29.171 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 14:26:29.171 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment fa229a47-8269-405f-a4eb-241278a933b2 to client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 14:26:29.171 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:26:29.172 INFO [standby-run-59] StandbyClientSyncExecution.java:196 Copying data segment 020e218c-9ae2-46af-a581-0272a3dd6fee from primary 14:26:29.172 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 for segment 020e218c-9ae2-46af-a581-0272a3dd6fee 14:26:29.173 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:26:29.173 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 14:26:29.173 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment 020e218c-9ae2-46af-a581-0272a3dd6fee for client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 14:26:29.173 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 448 sent to client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 14:26:29.173 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment 020e218c-9ae2-46af-a581-0272a3dd6fee to client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 14:26:29.174 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:26:29.938 INFO [standby-run-59] StandbyClientSyncExecution.java:82 updated head state successfully: true in 772ms. 14:26:29.939 DEBUG [standby-run-59] StandbyClient.java:157 Channel closed 14:26:29.939 DEBUG [proxy-worker-193] ForwardHandler.java:98 Connection to remote host closed 14:26:29.940 DEBUG [main] NetworkErrorProxy.java:116 Channel disconnected 14:26:29.940 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=-1, skip=-1,0 14:26:29.944 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 62875 14:26:29.945 DEBUG [proxy-worker-193] ForwardHandler.java:103 Group shut down 14:26:29.949 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 for current head 14:26:29.967 DEBUG [proxy-worker-194] ForwardHandler.java:88 Connected to remote host 14:26:29.968 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:62979 is allowed 14:26:29.968 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 14:26:29.968 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 14:26:29.969 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 14:26:29.969 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head 020e218c-9ae2-46af-a581-0272a3dd6fee.0000000c to client c63c6e17-a643-4a2f-81c9-5c34bb64b5f6 14:26:29.970 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 14:26:29.971 DEBUG [standby-run-60] StandbyClient.java:157 Channel closed 14:26:29.971 DEBUG [proxy-worker-194] ForwardHandler.java:98 Connection to remote host closed 14:26:29.976 DEBUG [proxy-worker-194] ForwardHandler.java:103 Group shut down 14:26:32.192 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:26:32.192 DEBUG [main] NetworkErrorProxy.java:128 Channel closed 14:26:32.195 DEBUG [main] NetworkErrorProxy.java:134 Boss group shut down 14:26:32.199 DEBUG [main] NetworkErrorProxy.java:139 Worker group shut down 14:26:32.199 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:26:32.200 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:26:32.200 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:26:32.203 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:26:32.215 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedIntermediateByte2 14:26:32.254 INFO [main] FileStore.java:466 TarMK closed: target\junit32718061797662336\junit6944884391267138663 14:26:32.285 INFO [main] FileStore.java:466 TarMK closed: target\junit32718061797662336\junit7700110466855248011 14:26:32.302 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit5613006805833384695\junit6235139555424570192, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:26:32.305 INFO [main] FileStore.java:240 TarMK opened: target\junit5613006805833384695\junit6235139555424570192 (mmap=false) 14:26:32.308 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit5613006805833384695\junit3218132789427840980, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:26:32.315 INFO [main] FileStore.java:240 TarMK opened: target\junit5613006805833384695\junit3218132789427840980 (mmap=false) 14:26:32.321 INFO [main] DataStoreTestBase.java:127 Test begin: testProxyFlippedStartByte 14:26:32.321 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:26:32.321 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:26:32.455 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0x52049772]' 14:26:32.457 DEBUG [main] StandbyServer.java:249 Binding was successful 14:26:32.496 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=0, skip=0,0 14:26:32.498 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 62983 14:26:33.071 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 6b9ee592-40d5-40e2-b47a-7d2396a098cc for current head 14:26:33.107 DEBUG [proxy-worker-201] ForwardHandler.java:88 Connected to remote host 14:26:33.107 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:63068 is allowed 14:26:33.108 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:26:33.108 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 6b9ee592-40d5-40e2-b47a-7d2396a098cc 14:26:33.109 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 6b9ee592-40d5-40e2-b47a-7d2396a098cc 14:26:33.109 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head f2d10e9d-0e05-481e-a03f-990fcf15bbd1.0000000c to client 6b9ee592-40d5-40e2-b47a-7d2396a098cc 14:26:33.110 WARN [standby-1] ExceptionHandler.java:37 Exception caught on client 6b9ee592-40d5-40e2-b47a-7d2396a098cc io.netty.handler.codec.compression.DecompressionException: Received COMPRESSED_DATA tag before STREAM_IDENTIFIER at io.netty.handler.codec.compression.SnappyFrameDecoder.decode(SnappyFrameDecoder.java:162) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) [netty-handler-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1342) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:934) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.14.Final.jar:4.1.14.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.14.Final.jar:4.1.14.Final] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] 14:26:38.073 ERROR [standby-run-61] StandbyClientSyncExecution.java:67 Unable to fetch remote head 14:26:38.073 DEBUG [standby-run-61] StandbyClient.java:157 Channel closed 14:26:38.073 DEBUG [main] NetworkErrorProxy.java:116 Channel disconnected 14:26:38.073 DEBUG [proxy-worker-201] ForwardHandler.java:98 Connection to remote host closed 14:26:38.073 INFO [main] NetworkErrorProxy.java:89 Starting proxy with flip=-1, skip=-1,0 14:26:38.073 DEBUG [main] NetworkErrorProxy.java:103 Bound on port 62983 14:26:38.073 DEBUG [proxy-worker-201] ForwardHandler.java:103 Group shut down 14:26:38.073 DEBUG [standby-2] GetHeadRequestEncoder.java:33 Sending request from client 6b9ee592-40d5-40e2-b47a-7d2396a098cc for current head 14:26:38.089 DEBUG [proxy-worker-202] ForwardHandler.java:88 Connected to remote host 14:26:38.089 DEBUG [primary-2] ClientFilterHandler.java:53 Client /127.0.0.1:63095 is allowed 14:26:38.089 DEBUG [primary-2] RequestDecoder.java:42 Parsed 'get head' message 14:26:38.089 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get head' received from client 6b9ee592-40d5-40e2-b47a-7d2396a098cc 14:26:38.089 DEBUG [primary-2] GetHeadRequestHandler.java:43 Reading head for client 6b9ee592-40d5-40e2-b47a-7d2396a098cc 14:26:38.089 DEBUG [primary-2] GetHeadResponseEncoder.java:36 Sending head f2d10e9d-0e05-481e-a03f-990fcf15bbd1.0000000c to client 6b9ee592-40d5-40e2-b47a-7d2396a098cc 14:26:38.089 DEBUG [standby-2] ResponseDecoder.java:82 Decoding 'get head' response 14:26:38.089 DEBUG [standby-run-62] StandbyClientSyncExecution.java:103 Found missing segment f2d10e9d-0e05-481e-a03f-990fcf15bbd1 14:26:38.089 DEBUG [standby-run-62] StandbyClientSyncExecution.java:124 Inspecting segment f2d10e9d-0e05-481e-a03f-990fcf15bbd1 14:26:38.089 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 6b9ee592-40d5-40e2-b47a-7d2396a098cc for references of segment f2d10e9d-0e05-481e-a03f-990fcf15bbd1 14:26:38.089 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:26:38.089 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment f2d10e9d-0e05-481e-a03f-990fcf15bbd1 for client 6b9ee592-40d5-40e2-b47a-7d2396a098cc 14:26:38.089 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment f2d10e9d-0e05-481e-a03f-990fcf15bbd1 to client 6b9ee592-40d5-40e2-b47a-7d2396a098cc 14:26:38.105 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:26:38.105 DEBUG [standby-run-62] StandbyClientSyncExecution.java:184 Found reference from f2d10e9d-0e05-481e-a03f-990fcf15bbd1 to 59456540-633c-4397-ae18-465f6da16da2 14:26:38.105 DEBUG [standby-run-62] StandbyClientSyncExecution.java:124 Inspecting segment 59456540-633c-4397-ae18-465f6da16da2 14:26:38.105 DEBUG [standby-2] GetReferencesRequestEncoder.java:33 Sending request from client 6b9ee592-40d5-40e2-b47a-7d2396a098cc for references of segment 59456540-633c-4397-ae18-465f6da16da2 14:26:38.105 DEBUG [primary-2] RequestDecoder.java:48 Parsed 'get references' message 14:26:38.105 DEBUG [primary-2] GetReferencesRequestHandler.java:39 Reading references of segment 59456540-633c-4397-ae18-465f6da16da2 for client 6b9ee592-40d5-40e2-b47a-7d2396a098cc 14:26:38.105 DEBUG [primary-2] GetReferencesResponseEncoder.java:34 Sending references of segment 59456540-633c-4397-ae18-465f6da16da2 to client 6b9ee592-40d5-40e2-b47a-7d2396a098cc 14:26:38.105 DEBUG [standby-2] ResponseDecoder.java:94 Decoding 'get references' response 14:26:38.105 INFO [standby-run-62] StandbyClientSyncExecution.java:196 Copying data segment 59456540-633c-4397-ae18-465f6da16da2 from primary 14:26:38.105 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 6b9ee592-40d5-40e2-b47a-7d2396a098cc for segment 59456540-633c-4397-ae18-465f6da16da2 14:26:38.105 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:26:38.105 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 6b9ee592-40d5-40e2-b47a-7d2396a098cc 14:26:38.105 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment 59456540-633c-4397-ae18-465f6da16da2 for client 6b9ee592-40d5-40e2-b47a-7d2396a098cc 14:26:38.105 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 192 sent to client 6b9ee592-40d5-40e2-b47a-7d2396a098cc 14:26:38.105 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment 59456540-633c-4397-ae18-465f6da16da2 to client 6b9ee592-40d5-40e2-b47a-7d2396a098cc 14:26:38.105 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:26:38.105 INFO [standby-run-62] StandbyClientSyncExecution.java:196 Copying data segment f2d10e9d-0e05-481e-a03f-990fcf15bbd1 from primary 14:26:38.105 DEBUG [standby-2] GetSegmentRequestEncoder.java:33 Sending request from client 6b9ee592-40d5-40e2-b47a-7d2396a098cc for segment f2d10e9d-0e05-481e-a03f-990fcf15bbd1 14:26:38.105 DEBUG [primary-2] RequestDecoder.java:45 Parsed 'get segment' message 14:26:38.105 DEBUG [primary-2] CommunicationObserver.java:120 Message 'get segment' received from client 6b9ee592-40d5-40e2-b47a-7d2396a098cc 14:26:38.105 DEBUG [primary-2] GetSegmentRequestHandler.java:39 Reading segment f2d10e9d-0e05-481e-a03f-990fcf15bbd1 for client 6b9ee592-40d5-40e2-b47a-7d2396a098cc 14:26:38.105 DEBUG [primary-2] CommunicationObserver.java:125 Segment with size 448 sent to client 6b9ee592-40d5-40e2-b47a-7d2396a098cc 14:26:38.105 DEBUG [primary-2] GetSegmentResponseEncoder.java:43 Sending segment f2d10e9d-0e05-481e-a03f-990fcf15bbd1 to client 6b9ee592-40d5-40e2-b47a-7d2396a098cc 14:26:38.105 DEBUG [standby-2] ResponseDecoder.java:86 Decoding 'get segment' response 14:26:38.120 INFO [standby-run-62] StandbyClientSyncExecution.java:82 updated head state successfully: true in 31ms. 14:26:38.120 DEBUG [standby-run-62] StandbyClient.java:157 Channel closed 14:26:38.120 DEBUG [proxy-worker-202] ForwardHandler.java:98 Connection to remote host closed 14:26:38.120 DEBUG [proxy-worker-202] ForwardHandler.java:103 Group shut down 14:26:40.459 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:26:40.459 DEBUG [main] NetworkErrorProxy.java:128 Channel closed 14:26:40.459 DEBUG [main] NetworkErrorProxy.java:134 Boss group shut down 14:26:40.459 DEBUG [main] NetworkErrorProxy.java:139 Worker group shut down 14:26:40.474 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:26:40.474 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:26:40.474 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:26:40.474 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:26:40.490 INFO [main] DataStoreTestBase.java:132 Test end: testProxyFlippedStartByte 14:26:40.844 INFO [main] FileStore.java:466 TarMK closed: target\junit5613006805833384695\junit3218132789427840980 14:26:40.891 INFO [main] FileStore.java:466 TarMK closed: target\junit5613006805833384695\junit6235139555424570192 14:26:40.907 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit4311685658024739499\junit4372250529482595471, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:26:40.907 INFO [main] FileStore.java:240 TarMK opened: target\junit4311685658024739499\junit4372250529482595471 (mmap=false) 14:26:40.907 INFO [main] FileStoreBuilder.java:340 Creating file store FileStoreBuilder{version=1.8-SNAPSHOT, directory=target\junit4311685658024739499\junit482265720534482259, blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore], maxFileSize=1, segmentCacheSize=0, stringCacheSize=0, templateCacheSize=0, stringDeduplicationCacheSize=15000, templateDeduplicationCacheSize=3000, nodeDeduplicationCacheSize=1, memoryMapping=false, gcOptions=SegmentGCOptions{paused=false, estimationDisabled=false, gcSizeDeltaEstimation=1073741824, retryCount=5, forceTimeout=60, retainedGenerations=2, gcType=FULL}} 14:26:40.922 INFO [main] FileStore.java:240 TarMK opened: target\junit4311685658024739499\junit482265720534482259 (mmap=false) 14:26:40.922 INFO [main] DataStoreTestBase.java:127 Test begin: testSyncBigBlob 14:26:40.922 INFO [main] SegmentNodeStore.java:120 Creating segment node store SegmentNodeStoreBuilder{blobStore=DataStore backed BlobStore [org.apache.jackrabbit.core.data.FileDataStore]} 14:26:40.922 INFO [main] LockBasedScheduler.java:155 Initializing SegmentNodeStore with the commitFairLock option enabled. 14:27:12.577 WARN [main] Slf4JLogger.java:146 Unknown channel option 'TCP_NODELAY' for channel '[id: 0xcccf4f88]' 14:27:12.577 DEBUG [main] StandbyServer.java:249 Binding was successful 14:27:12.577 DEBUG [standby-1] GetHeadRequestEncoder.java:33 Sending request from client 44dc1176-74bb-46bd-9e73-c945709bcd3c for current head 14:27:12.577 DEBUG [primary-1] ClientFilterHandler.java:53 Client /127.0.0.1:63173 is allowed 14:27:12.593 DEBUG [primary-1] RequestDecoder.java:42 Parsed 'get head' message 14:27:12.594 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get head' received from client 44dc1176-74bb-46bd-9e73-c945709bcd3c 14:27:12.594 DEBUG [primary-1] GetHeadRequestHandler.java:43 Reading head for client 44dc1176-74bb-46bd-9e73-c945709bcd3c 14:27:12.970 DEBUG [primary-1] GetHeadResponseEncoder.java:36 Sending head e7a7b225-cd8b-40d7-aec1-0e219376bcf6.00000005 to client 44dc1176-74bb-46bd-9e73-c945709bcd3c 14:27:12.970 DEBUG [standby-1] ResponseDecoder.java:82 Decoding 'get head' response 14:27:12.970 DEBUG [standby-run-63] StandbyClientSyncExecution.java:103 Found missing segment e7a7b225-cd8b-40d7-aec1-0e219376bcf6 14:27:12.970 DEBUG [standby-run-63] StandbyClientSyncExecution.java:124 Inspecting segment e7a7b225-cd8b-40d7-aec1-0e219376bcf6 14:27:12.970 DEBUG [standby-1] GetReferencesRequestEncoder.java:33 Sending request from client 44dc1176-74bb-46bd-9e73-c945709bcd3c for references of segment e7a7b225-cd8b-40d7-aec1-0e219376bcf6 14:27:12.970 DEBUG [primary-1] RequestDecoder.java:48 Parsed 'get references' message 14:27:12.970 DEBUG [primary-1] GetReferencesRequestHandler.java:39 Reading references of segment e7a7b225-cd8b-40d7-aec1-0e219376bcf6 for client 44dc1176-74bb-46bd-9e73-c945709bcd3c 14:27:13.095 DEBUG [primary-1] GetReferencesResponseEncoder.java:34 Sending references of segment e7a7b225-cd8b-40d7-aec1-0e219376bcf6 to client 44dc1176-74bb-46bd-9e73-c945709bcd3c 14:27:13.096 DEBUG [standby-1] ResponseDecoder.java:94 Decoding 'get references' response 14:27:13.097 INFO [standby-run-63] StandbyClientSyncExecution.java:196 Copying data segment e7a7b225-cd8b-40d7-aec1-0e219376bcf6 from primary 14:27:13.098 DEBUG [standby-1] GetSegmentRequestEncoder.java:33 Sending request from client 44dc1176-74bb-46bd-9e73-c945709bcd3c for segment e7a7b225-cd8b-40d7-aec1-0e219376bcf6 14:27:13.099 DEBUG [primary-1] RequestDecoder.java:45 Parsed 'get segment' message 14:27:13.100 DEBUG [primary-1] CommunicationObserver.java:120 Message 'get segment' received from client 44dc1176-74bb-46bd-9e73-c945709bcd3c 14:27:13.100 DEBUG [primary-1] GetSegmentRequestHandler.java:39 Reading segment e7a7b225-cd8b-40d7-aec1-0e219376bcf6 for client 44dc1176-74bb-46bd-9e73-c945709bcd3c 14:27:13.100 DEBUG [primary-1] CommunicationObserver.java:125 Segment with size 192 sent to client 44dc1176-74bb-46bd-9e73-c945709bcd3c 14:27:13.100 DEBUG [primary-1] GetSegmentResponseEncoder.java:43 Sending segment e7a7b225-cd8b-40d7-aec1-0e219376bcf6 to client 44dc1176-74bb-46bd-9e73-c945709bcd3c 14:27:13.100 DEBUG [standby-1] ResponseDecoder.java:86 Decoding 'get segment' response 14:27:13.896 INFO [standby-run-63] StandbyClientSyncExecution.java:82 updated head state successfully: true in 926ms. 14:27:13.897 DEBUG [standby-run-63] StandbyClient.java:157 Channel closed 14:27:16.133 DEBUG [main] StandbyClientSync.java:277 Group shut down 14:27:16.133 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:27:16.133 DEBUG [main] StandbyServer.java:220 Channel disconnected 14:27:16.133 DEBUG [main] StandbyServer.java:231 Boss group shut down 14:27:16.133 DEBUG [main] StandbyServer.java:237 Worker group shut down 14:27:16.133 INFO [main] DataStoreTestBase.java:132 Test end: testSyncBigBlob 14:27:16.180 INFO [main] FileStore.java:466 TarMK closed: target\junit4311685658024739499\junit482265720534482259 14:27:16.218 INFO [main] FileStore.java:466 TarMK closed: target\junit4311685658024739499\junit4372250529482595471