Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1078

Under heavy load, log cache reads return "Op in future"

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Private Beta
    • Fix Version/s: 1.6.0
    • Component/s: consensus
    • Labels:
      None

      Description

      JD accidentally configured botl80 so that all nodes were writing to a single tablet (talk about a stress test!) I see the following warning occasionally in the leader logs:

      W0827 11:34:04.051275 55950 consensus_queue.cc:272] T d6ff74fb04454712873abd0d2328e59b P 668314723deb4a818cc9a43eba51073c [LEADER]: The logs necessary to catch up peer 600e62435ca24425b29c00d9726b78be have been garbage collected. The follower will never be able to catch up (Not found: op in future). Instructing remote peer to remotely bootstrap.

      Calling this a blocker because, if we couple this with the "automatically re-bootstrap nodes if they fall behind", we'll be accidentally deleting tablets when this happens (no good!)

        Attachments

          Activity

            People

            • Assignee:
              tlipcon Todd Lipcon
              Reporter:
              tlipcon Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: