[HDDS-2697] SCM log is flooded with block deletion txId mismatch messages

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Component/s: SCM
• Labels: TriagePending ozone-jira-volunteer ozone-moved-out-critical
• Target Version/s: 2.0.0
• Assignee: Abhishek Pal
• Reporter: István Fajth
• Created: 09/Dec/19 16:08
• Updated: 17/Oct/24 14:56
• Resolved: 17/Oct/24 14:56

Description

When you run Hive queries on the cluster, but I think this is true for other MapReduce stuff as well, then the interim and temporary data is created and deleted quite often.
This leads to the flood of similar messages in the SCM log:

2019-12-07 05:00:41,112 INFO org.apache.hadoop.hdds.scm.block.SCMBlockDeletingService: Block deletion txnID mismatch in datanode e590d08a-4a4e-428a-82e8-80f7221f639e for containerID 307. Datanode delete txnID: 25145, SCM txnID: 25148

Either we need to decrease the log level of this messages, or we need to get rid of the cause of the message. In a single log file I see over 21k lines containing this message from ~37k lines of log.