The issue relates to incorrect IOOM handling that can occurs on Tx prepare\commit\rollback and can be reproduced if persistence enabled and Tx state logging into WAL enabled.
This affects MVCC tx as it always log it's state into WAL and non-MVCC Tx with enabled WAL logging via setting IGNITE_WAL_LOG_TX_RECORDS system property.
We have to check and fix if tx finish methods handle RuntimeExceptions in proper way.
Good start is to force throw RuntimeException from tm().mvccPrepare() and tm().mvccFinish() methods, and check if DhtFinishFuture done correctly with exception rather then (re)throwing exception bypassing failure handler.
The goal is to make IoomFailureHandlerTest passed after runtime failures during Tx commit\prepare\rollback.