[HDDS-10706] OMDoubleBuffer error while creating snapshot

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: OM, Snapshot
• Labels: ozone-snapshot
• Assignee: Swaminathan Balachandran
• Reporter: Jyotirmoy Sinha
• Created: 17/Apr/24 07:05
• Updated: 22/Apr/24 16:31

Description

Scenario -

• Generate data over parallel threads over various volume/buckets
• Perform parallel snapshot create/delete/list operations over above buckets
• Perform parallel snapdiff operations over each bucket
• Perform parallel read operations of snapshot contents
• Introduce OM and cluster restarts in between along with DN decommissioning and balancer restarts.

OM Leader error stacktrace -

2024-04-16 22:51:39,385 ERROR [OMDoubleBufferFlushThread]-org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer: Terminating with exit status 1: During flush to DB encountered error in OMDoubleBuffer flush thread OMDoubleBufferFlushThread when handling OMRequest: cmdType: CreateSnapshot
traceID: ""
success: true
status: OK
CreateSnapshotResponse {
  snapshotInfo {
    snapshotID {
      mostSigBits: -8880365051952676788
      leastSigBits: -7918193529622336024
    }
    name: "snap339"
    volumeName: "voltest21713265900"
    bucketName: "buck1"
    snapshotStatus: SNAPSHOT_ACTIVE
    creationTime: 1713333097858
    deletionTime: 18446744073709551615
    pathPreviousSnapshotID {
      mostSigBits: -3527873859313710896
      leastSigBits: -6772710914652776790
    }
    globalPreviousSnapshotID {
      mostSigBits: -8910008813791655688
      leastSigBits: -8609017308136858185
    }
    snapshotPath: "voltest21713265900/buck1"
    checkpointDir: "-84c29b09-1291-404c-921c-ecefec106de8"
    dbTxSequenceNumber: 59906687
    deepClean: true
    sstFiltered: false
  }
}

java.io.IOException: Rocks Database is closed
        at org.apache.hadoop.hdds.utils.db.RocksDatabase.acquire(RocksDatabase.java:439)
        at org.apache.hadoop.hdds.utils.db.RocksDatabase.newIterator(RocksDatabase.java:777)
        at org.apache.hadoop.hdds.utils.db.RDBTable.iterator(RDBTable.java:232)
        at org.apache.hadoop.hdds.utils.db.TypedTable.iterator(TypedTable.java:418)
        at org.apache.hadoop.hdds.utils.db.TypedTable.iterator(TypedTable.java:55)
        at org.apache.hadoop.ozone.om.OmSnapshotManager.deleteKeysFromDelKeyTableInSnapshotScope(OmSnapshotManager.java:573)
        at org.apache.hadoop.ozone.om.OmSnapshotManager.createOmSnapshotCheckpoint(OmSnapshotManager.java:444)
        at org.apache.hadoop.ozone.om.response.snapshot.OMSnapshotCreateResponse.addToDBBatch(OMSnapshotCreateResponse.java:81)
        at org.apache.hadoop.ozone.om.response.OMClientResponse.checkAndUpdateDB(OMClientResponse.java:73)
        at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$5(OzoneManagerDoubleBuffer.java:382)
        at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatchWithTrace(OzoneManagerDoubleBuffer.java:220)
        at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatch(OzoneManagerDoubleBuffer.java:381)
        at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushBatch(OzoneManagerDoubleBuffer.java:324)
        at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushCurrentBuffer(OzoneManagerDoubleBuffer.java:297)
        at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:262)
        at java.lang.Thread.run(Thread.java:748)