Uploaded image for project: 'ZooKeeper'
  1. ZooKeeper
  2. ZOOKEEPER-1090

Race condition while taking snapshot can lead to not restoring data tree correctly

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      I think I have found a bug in the snapshot mechanism.

      The problem occurs because dt.lastProcessedZxid is not synchronized (or rather set before the data tree is modified):

      FileTxnSnapLog:

          public void save(DataTree dataTree,
                  ConcurrentHashMap<Long, Integer> sessionsWithTimeouts)
              throws IOException {
              long lastZxid = dataTree.lastProcessedZxid;
              LOG.info("Snapshotting: " + Long.toHexString(lastZxid));
              File snapshot=new File(
                      snapDir, Util.makeSnapshotName(lastZxid));
              snapLog.serialize(dataTree, sessionsWithTimeouts, snapshot);   <=== the Datatree may not have the modification for lastProcessedZxid
          }
      

      DataTree:

          public ProcessTxnResult processTxn(TxnHeader header, Record txn) {
              ProcessTxnResult rc = new ProcessTxnResult();
      
              String debug = "";
              try {
                  rc.clientId = header.getClientId();
                  rc.cxid = header.getCxid();
                  rc.zxid = header.getZxid();
                  rc.type = header.getType();
                  rc.err = 0;
                  if (rc.zxid > lastProcessedZxid) {
                      lastProcessedZxid = rc.zxid;
                  }
                  [...modify data tree...]           
       }
      

      The lastProcessedZxid must be set after the modification is done.

      As a result, if server crashes after taking the snapshot (and the snapshot does not contain change corresponding to lastProcessedZxid) restore will not restore the data tree correctly:

      public long restore(DataTree dt, Map<Long, Integer> sessions,
                  PlayBackListener listener) throws IOException {
              snapLog.deserialize(dt, sessions);
              FileTxnLog txnLog = new FileTxnLog(dataDir);
              TxnIterator itr = txnLog.read(dt.lastProcessedZxid+1); <=== Assumes lastProcessedZxid is deserialized
       }
      

      I have had offline discussion with Ben and Camille on this. I will be posting the discussion shortly.

      Attachments

        1. ZOOKEEPER-1090
          7 kB
          Vishal Kher

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            vishalmlst Vishal Kher
            vishalmlst Vishal Kher
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment