[HDDS-8786] Encounter error of OzoneManagerDoubleBuffer Terminating during snapshot creation

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Component/s: Ozone Manager
• Labels: ozone-snapshot
• Assignee: Hemant Kumar
• Reporter: Dave Teng
• Created: 08/Jun/23 03:26
• Updated: 22/Jun/23 20:41
• Resolved: 22/Jun/23 20:41

Description

When creating snapshot, encountered following error in OM:

2023-06-08 02:01:14,771 [OMDoubleBufferFlushThread] ERROR org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer: Terminating with exit statu
s 1: During flush to DB encountered error in OMDoubleBuffer flush thread OMDoubleBufferFlushThread when handling OMRequest: cmdType: CreateSnaps
hot
traceID: ""
success: true
status: OK
CreateSnapshotResponse {
  snapshotInfo {
    snapshotID: "3ce30204-da71-41ad-9e78-607c688e8c5c"
    name: "snap5670"
    volumeName: "vol2"
    bucketName: "bucket2"
    snapshotStatus: SNAPSHOT_ACTIVE
    creationTime: 1686189673557
    deletionTime: 18446744073709551615
    pathPreviousSnapshotID: "180bd9f4-a83e-4bfb-97b3-a19d0eef0450"
    globalPreviousSnapshotID: "180bd9f4-a83e-4bfb-97b3-a19d0eef0450"
    snapshotPath: "vol2/bucket2"
    checkpointDir: "-3ce30204-da71-41ad-9e78-607c688e8c5c"
    dbTxSequenceNumber: 3269138
  }
}

java.io.IOException: Rocks Database is closed
        at org.apache.hadoop.hdds.utils.db.RocksDatabase.assertClose(RocksDatabase.java:442)
        at org.apache.hadoop.hdds.utils.db.RocksDatabase.newIterator(RocksDatabase.java:827)
        at org.apache.hadoop.hdds.utils.db.RDBTable.iterator(RDBTable.java:194)
        at org.apache.hadoop.hdds.utils.db.TypedTable.iterator(TypedTable.java:361)
        at org.apache.hadoop.ozone.om.OmSnapshotManager.deleteKeysFromDelKeyTableInSnapshotScope(OmSnapshotManager.java:573)
        at org.apache.hadoop.ozone.om.OmSnapshotManager.createOmSnapshotCheckpoint(OmSnapshotManager.java:394)
        at org.apache.hadoop.ozone.om.response.snapshot.OMSnapshotCreateResponse.addToDBBatch(OMSnapshotCreateResponse.java:81)
        at org.apache.hadoop.ozone.om.response.OMClientResponse.checkAndUpdateDB(OMClientResponse.java:73)
        at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$5(OzoneManagerDoubleBuffer.java:414)
at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatchWithTrace(OzoneManagerDoubleBuffer.java:237)
        at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatch(OzoneManagerDoubleBuffer.java:412)
        at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushBatch(OzoneManagerDoubleBuffer.java:333)
        at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushCurrentBuffer(OzoneManagerDoubleBuffer.java:312)
        at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:277)
        at java.lang.Thread.run(Thread.java:748)