[IGNITE-8429] Unexpected error during incorrect WAL segment decompression, causes node termination.

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.5
• Fix Version/s: 2.5
• Component/s: persistence
• Labels: WAL
• Assignee: Ivan Rakov
• Reporter: Ivan Daschinsky
• Created: 03/May/18 08:31
• Updated: 23/Jul/19 17:34
• Resolved: 08/May/18 12:22

Description

File decompressor failure due to incorrect (zero-length) archived segment.

2018-04-30 00:00:02.811 [ERROR][wal-file-decompressor%DPL_GRID%DplGridNodeName][org.apache.ignite.Ignite] Critical system error detected. Will be handled accordingly to configured handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler, failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.IllegalStateException: Thread wal-file-decompressor%DPL_GRID%DplGridNodeName is terminated unexpectedly]]
java.lang.IllegalStateException: Thread wal-file-decompressor%DPL_GRID%DplGridNodeName is terminated unexpectedly
at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDecompressor.run(FileWriteAheadLogManager.java:2104)
2018-04-30 00:00:02.812 [ERROR][wal-file-decompressor%DPL_GRID%DplGridNodeName][org.apache.ignite.Ignite] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.IllegalStateException: Thread wal-file-decompressor%DPL_GRID%DplGridNodeName is terminated unexpectedly]]

touch 0000000000000754.wal
zip 0000000000000754.wal.zip 0000000000000754.wal
ls -l
rw-rw-r- 1 dmitriy dmitriy 0 май 1 16:40 0000000000000754.wal
rw-rw-r- 1 dmitriy dmitriy 190 май 1 16:46 0000000000000754.wal.zip

Archive: /tmp/temp/0000000000000754.wal.zip
Length Method Size Cmpr Date Time CRC-32 Name
-------- ------ ------- ---- ---------- ----- -------- ----
0 Stored 0 0% 2018-05-01 16:40 00000000 0000000000000754.wal
-------- ------- — -------
0 0 0% 1 file

We should softly handle this situation: print message in log and continue the decompression with next segment.
We also should handle "skipped" segments and don't delete them in deleteObsoleteRawSegments().

Issue Links

• causes: IGNITE-8453 FileDecompressor may access HashMap without proper synchronization
• relates to: IGNITE-8393 Unexpected error during WAL compression java.io.EOFException: EOF at position [0] expected to read [29] bytes
• links to: GitHub Pull Request #3955
• links to: GitHub Pull Request #3955