https://issues.apache.org/jira This file is an XML representation of an issue en-uk 6.3.4 6332 15-08-2014 https://issues.apache.org/jira/browse/OAK-5446 Jackrabbit Oak <p>Fighting with cluster nodes losing their lease and shutting down oak-core in a cloud environment. For reasons unknown at this point in time, the whole process seems to skip about two minutes of real time.</p> <p>This is a situation from which oak currently does not recover. Code analysis shows that <tt>ClusterNodeInfo</tt> is handed the <tt>LeaseCheckDocumentStoreWrapper</tt> instance to use as store. This is fatal since any action the <tt>renewLease()</tt> tries to do will first invoke the <tt>performLeaseCheck()</tt>. The lease check will, when the <tt>FailureMargin</tt> is reached, <em>stall the renewLease() thread</em> for 5 retry attempts and then declare the lease to be lost.</p> <p>The <tt>ClusterNodeInfo</tt> should instead be using the "real" <tt>DocumentStore</tt>, not the wrapped one, IMO.</p> OAK-5446

leaseUpdateThread might be blocked by leaseUpdateCheck

Bug Major Open Unresolved Julian Reschke Stefan Eissing candidate_oak_1_4 Thu, 12 Jan 2017 09:58:36 +0000 Thu, 26 Jan 2017 16:58:54 +0000 1.4 1.5.14 1.6 core 0 3 <p>Would be good to have a unit test (looking now).</p> <p>FWIW, another way to avoid this would be to special case the CLUSTER collection in the <tt>LeaseCheckDocumentStoreWrapper</tt>.</p> <p>I agree, there should be a test. Looks like <a href="https://issues.apache.org/jira/browse/OAK-3399" title="5sec retry loop before declaring lease failure (was: as a last resort try to update it there)" class="issue-link" data-issue-key="OAK-3399"><del>OAK-3399</del></a> didn't have one. </p> <p>I'll try to come up with a test case.</p> <p>Hmmm.</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> public DocumentNodeStore(DocumentMK.Builder builder) { this.blobStore = builder.getBlobStore(); this.statisticsProvider = builder.getStatisticsProvider(); this.nodeStoreStatsCollector = builder.getNodeStoreStatsCollector(); if (builder.isUseSimpleRevision()) { this.simpleRevisionCounter = new AtomicInteger(0); } DocumentStore s = builder.getDocumentStore(); if (builder.getTiming()) { s = new TimingDocumentStoreWrapper(s); } if (builder.getLogging()) { s = new LoggingDocumentStoreWrapper(s); } if (builder.getReadOnlyMode()) { s = ReadOnlyDocumentStoreWrapperFactory.getInstance(s); readOnlyMode = true; } else { readOnlyMode = false; } this.executor = builder.getExecutor(); this.clock = builder.getClock(); int cid = builder.getClusterId(); cid = Integer.getInteger("oak.documentMK.clusterId", cid); if (readOnlyMode) { clusterNodeInfo = ClusterNodeInfo.getReadOnlyInstance(s); } else { clusterNodeInfo = ClusterNodeInfo.getInstance(s, cid); } // TODO we should ensure revisions generated from now on // are never "older" than revisions already in the repository for // this cluster id cid = clusterNodeInfo.getId(); if (builder.getLeaseCheck()) { s = new LeaseCheckDocumentStoreWrapper(s, clusterNodeInfo); clusterNodeInfo.setLeaseFailureHandler(builder.getLeaseFailureHandler()); } else { clusterNodeInfo.setLeaseCheckDisabled(true); } </pre> </div></div> <p>So the <tt>DocumentStore</tt> is passed into <tt>ClusterNodeInfo</tt> <b>before</b> it get's wrapped into a <tt>LeaseCheckDocumentStoreWrapper</tt>, no?</p> <p>But then, the lease update thread also does:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre> // then, independently if the lease had to be updated or not, check // the status: if (nodeStore.updateClusterState()) { // then inform the discovery lite listener - if it is registered nodeStore.signalClusterStateChange(); } </pre> </div></div> <p>and, AFAICT, that indeed would go through the lease checking wrapper...</p> <p>It's not hard to fix this, but it's not entirely clear what the best way is... </p> <ul class="alternate" type="square"> <li>decouple updateClusterState from lease renew thread &#8211; probably good because the latter is timing critical</li> <li>make sure LeaseChecker only delays calls to a whitelisted set of collections (just NODES?)</li> <li>make sure that (some?) background threads always bypass the LeaseCheckWrapper</li> </ul> <p>Also: is it really intentional that <tt>updateClusterState</tt> reads all cluster node infos once per second??? </p> <p><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=egli" class="user-hover" rel="egli">Stefan Egli</a> <a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=mreutegg" class="user-hover" rel="mreutegg">Marcel Reutegger</a> - feedback appreciated.</p> <p>Proposed patch.</p> <blockquote><p>Code analysis shows that ClusterNodeInfo is handed the LeaseCheckDocumentStoreWrapper instance to use as store. </p></blockquote> <p>I second <a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=reschke" class="user-hover" rel="reschke">Julian Reschke</a>'s finding that ClusterNodeInfo gets the 'real' DocumentStore, not the lease wrapper. <a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=stefan.eissing" class="user-hover" rel="stefan.eissing">Stefan Eissing</a>, do you disagree?</p> <blockquote><p>But then, the lease update thread also does [<tt>nodeStore.updateClusterState()</tt>]</p></blockquote> <p>good point, that one goes via the LeaseCheckDocumentStoreWrapper, while it probably should not, agreed.</p> <blockquote><p>It's not hard to fix this, but it's not entirely clear what the best way is... </p></blockquote> <p>I would try to come up with the least intrusive change. Which would mean to leave the lease wrapper as is (ie no whitelist or bypassing special cases). Ideally <tt>updateClusterState</tt> would be aware of the lease wrapper and pass the underlying (real) documentStore instead. Perhaps the DocumentNodeStore could keep track of that real documentStore for use in <tt>updateClusterState</tt>?</p> <blockquote><p>Also: is it really intentional that updateClusterState reads all cluster node infos once per second??? </p></blockquote> <p>Yes, that is intentional. It is part of discovery lite where each instance checks the state of all the cluster node infos once a second to notice a crashing instance asap. We can discuss if the 1/second resolution is too much or if it should be done in a separate thread, but the discovery lite mechanism is based on this.</p> <blockquote><p>I second Julian Reschke's finding that ClusterNodeInfo gets the 'real' DocumentStore, not the lease wrapper. Stefan Eissing, do you disagree?</p></blockquote> <p>No, I agree. I saw in the logs that the lease update thread used the LeaseWrapper and was blocked. But the call is in the <tt>updateClusterState()</tt> that caused this.</p> <p>So, we are all in agree and that this should be changed. As to how, I have no strong opinion. Perhaps the whole <tt>updateClusterState</tt> should go into the <tt>ClusterNodeInfo</tt>?</p> <p>Reminder: there's a patch available. The IMHO most important part is to separate the two operations into separate background threads.</p> <p>+1 for the patch. Separating the cluster update from the lease update - even though these two operation are about a related concern - is a good thing. (sorry, missed reviewing the patch previously)</p> <p><a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=egli" class="user-hover" rel="egli">Stefan Egli</a> &amp; <a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=mreutegg" class="user-hover" rel="mreutegg">Marcel Reutegger</a> and we are ok with this to be done before we branch 1.6, right?</p> <blockquote><p>we are ok with this to be done before we branch 1.6</p></blockquote> <p>it is a change in a quite central part, so the question is if this is indeed a blocker for 1.6 or if it can't wait... We'd have to <del>thoroughly</del> test the fix well.</p> <p>Added a simplistic unit test that covers the retry loop (<span class="nobr"><a href="https://issues.apache.org/jira/secure/attachment/12849312/12849312_OAK-5446.testcase" title="OAK-5446.testcase attached to OAK-5446">OAK-5446.testcase^{<img class="rendericon" src="https://issues.apache.org/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span>) - might be good enough as a start?</p> <p>Verified that the test tests the right thing by setting MAX_RETRY_SLEEPS_BEFORE_LEASE_FAILURE = 0 in ClusterNodeInfo (in which case the test fails).</p> <p>Proposal: add the test as part of <a href="https://issues.apache.org/jira/browse/OAK-3399" title="5sec retry loop before declaring lease failure (was: as a last resort try to update it there)" class="issue-link" data-issue-key="OAK-3399"><del>OAK-3399</del></a> (after removing trailing ws <img class="emoticon" src="https://issues.apache.org/jira/images/icons/emoticons/smile.gif" height="16" width="16" align="absmiddle" alt="" border="0"/>.</p> <p>We still don't have a test that simulates the issue described in <b>this</b> ticket, though.</p> <p>Modified test that delays the read from clusterNodes and indeed reproduces the issue.</p> <p>ack, I'll add the test case to <a href="https://issues.apache.org/jira/browse/OAK-3399" title="5sec retry loop before declaring lease failure (was: as a last resort try to update it there)" class="issue-link" data-issue-key="OAK-3399"><del>OAK-3399</del></a> (in trunk and 1.4 branch) and will look into how we can simulate a 'VM freeze' during lease update..</p> <blockquote><p>Modified test that delays the read from clusterNodes and indeed reproduces the issue.</p></blockquote> <p>even better <img class="emoticon" src="https://issues.apache.org/jira/images/icons/emoticons/smile.gif" height="16" width="16" align="absmiddle" alt="" border="0"/> - can I leave the patch with you, <a href="https://issues.apache.org/jira/secure/ViewProfile.jspa?name=reschke" class="user-hover" rel="reschke">Julian Reschke</a>?</p> <p>just one comment: maybe we should have two flavours of the test: one with the delay and one without - as both cases seem useful.</p> <p>While the test currently proves that there is a problem, I'm not totally sure that the Thread.sleep is correct here &#8211; shouldn't I use the VirtualClock?</p> <p>I think actually delaying is correct, as a) the background thread doesn't use the virtual clock anyway and b) you want to simulate a 'vm freeze' for which you need to actually sleep.<br/> But I think the delaying should only happen after the first successful renewal (to keep the idea of that test method) - so I've added a variant which enables delaying after forwarding the clock (<span class="nobr"><a href="https://issues.apache.org/jira/secure/attachment/12849324/12849324_OAK-5446.testcase.v3" title="OAK-5446.testcase.v3 attached to OAK-5446">OAK-5446.testcase.v3^{<img class="rendericon" src="https://issues.apache.org/jira/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span>)</p> <p>...guess what: in the proposed patch I <b>copied</b> (instead of <b>moved</b>) the cluster update check... <img class="emoticon" src="https://issues.apache.org/jira/images/icons/emoticons/smile.gif" height="16" width="16" align="absmiddle" alt="" border="0"/></p> <p>oups, yes, that would not have helped then..</p> <p>See <a href="http://svn.apache.org/r1780424" class="external-link" rel="nofollow">r1780424</a> - for some I reason I currently can not resolve this ticket though.</p> <p>Note I had to clone the issue to work around a JIRA problem.</p> Cloners OAK-5528 Reference OAK-3399 OAK-5523 Attachment count 4.0 Date of First Response Thu, 12 Jan 2017 14:46:22 +0000 Global Rank 9223372036854775807 Last public comment date 8 minutes ago Rank 0|i38m27: Rank (Obsolete) 9223372036854775807