Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1387

Scanner gets into tight loop followed by long sleep when leader TS is down

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 0.7.0
    • 0.8.0
    • client
    • None

    Description

      I spent some time looking into linked_list-test flakiness and found the following really bad behavior:

      • the leader TS of a tablet goes down
      • the other two replicas are up but in a state that prevents scanning (eg bootstrapping)
      • the scanner tries to read the two "bootstrapping" tablets, and gets some error which ends up adding the replicas to the scanner's blacklist.
      • it tries to read from the down TS, and gets "connection refused". We handle this slightly differently since it's a TS-wide error, and we mark the TS as down in the meta cache. We currently do not add it to the scanner's blacklist. We also do not do any backoff/sleep.
      • on the next LookupTablet call, we see that the last-known leader is marked down, and decide that we should refresh locations from the master.
        • when we get the response back from the master, we clear the 'failed' flag on the replica. (I'm not sure this is justifiable, but it's the current policy)
      • we then try to select a TS for the scanner again, and since the down machine is no longer marked as "failed", and not in the blacklist, we select it again.
      • the loop continues from the top

      Because there is no sleeping involved here, we end up doing hundreds or thousands of round trips to the master here as we wait for the leader to change. This would be somewhat bad on its own, but in fact the problem is even worse: when the master finally learns of a new leader, the cycle breaks. Meanwhile, we had increased the 'attempts' count to a very high value, and then our backoff code decides it's a good idea to sleep for 800 seconds (regardless of deadline, etc). This causes linked_list-test to time out and probably would cause lots of problems for real use cases as well.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            tlipcon Todd Lipcon
            tlipcon Todd Lipcon
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment