Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2476

Kudu restart creates many tombstone tablets

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Information Provided
    • Affects Version/s: 1.5.0
    • Fix Version/s: n/a
    • Component/s: None
    • Labels:
      None

      Description

      After changing chronyc conf and restarting on all nodes the chronyd daemon, all the Kudu servers exited unexpectedly. Therefore I did restart all the Kudu nodes (tablet servers and masters) and when they came up, many tablets were in initialized state and many tablets ended in a tombstone state.

      Flags
      Live Tablets
      Summary
      Status Count Percentage
      BOOTSTRAPPING 4 0.50
      INITIALIZED 117 14.68
      RUNNING 676 84.82
      Total 797

      The tables consistency seems ok after the restart, but I have two questions:

      • is it ok that when the timesync changes all the nodes suddenly crashes?
      • is it ok that after Kudu service restart the Kudu tablets goes crazy and starts to send accross the network lot of data, "syncing" up, which takes ~5-10minutes (800tablets/kudu tablet server)

      Shouldnt be the behaviour that the Kudu tablet server waits a little bit during the restart and then starts to replicate the data?

        Attachments

          Activity

            People

            • Assignee:
              wdberkeley Will Berkeley
              Reporter:
              farkastfbic Tomas Farkas
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: