[HDDS-6934] OM crashed with OzoneManagerDoubleBuffer

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.3.0
• Component/s: Ozone Filesystem
• Labels: pull-request-available
• Target Version/s: 1.3.0
• Assignee: Rakesh Radhakrishnan
• Reporter: Soumitra Sulav
• Created: 22/Jun/22 19:44
• Updated: 28/Jun/22 14:44
• Resolved: 28/Jun/22 14:44

Description

Steps followed while we encounter this issue.

• A test bucket was created with FILE_SYSTEM_OPTIMIZED as the layout type.
  ofs://ozone1/prefix94441/layout97486/
• Multiple keys and directories where created in this bucket (1000+).
• ListStatus operation was done to verify the sorted list.
• Once validated the bucket was recursively deleted using FS api.

RUNNING: /opt/cloudera/parcels/CDH/bin/ozone fs -rm -r -skipTrash 'ofs://ozone1/prefix94441/layout97486/'
Deleted ofs://ozone1/prefix94441/layout97486
Exit Code: 0

Post above operation we see OzoneManagerDoubleBuffer exception as below stacktrace

2022-06-22 11:53:30,907 ERROR org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer: Terminating with exit status 2: OMDoubleBuffer flush thread OMDoubleBufferFlushThread encountered Throwable error
java.lang.NullPointerException
	at org.apache.hadoop.ozone.om.OmMetadataManagerImpl.getBucketId(OmMetadataManagerImpl.java:1482)
	at org.apache.hadoop.ozone.om.response.key.OMKeysDeleteResponseWithFSO.addToDBBatch(OMKeysDeleteResponseWithFSO.java:67)
	at org.apache.hadoop.ozone.om.response.key.OMKeysDeleteResponse.checkAndUpdateDB(OMKeysDeleteResponse.java:70)
	at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$2(OzoneManagerDoubleBuffer.java:283)
	at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.addToBatchWithTrace(OzoneManagerDoubleBuffer.java:228)
	at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.lambda$1(OzoneManagerDoubleBuffer.java:281)
	at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
	at org.apache.hadoop.ozone.om.ratis.OzoneManagerDoubleBuffer.flushTransactions(OzoneManagerDoubleBuffer.java:277)
	at java.base/java.lang.Thread.run(Thread.java:834)

Issue Links

• is related to: HDDS-6768 [FSO] Add Volume and Bucket ID to the key path for FSO Objects
• links to: GitHub Pull Request #3544