[IGNITE-8393] Unexpected error during WAL compression java.io.EOFException: EOF at position [0] expected to read [29] bytes

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.5
• Component/s: persistence
• Labels: WAL
• Assignee: Ivan Rakov
• Reporter: Sergey Filatov
• Created: 26/Apr/18 09:59
• Updated: 23/Jul/19 17:34
• Resolved: 27/Apr/18 13:43

Description

In WAL segment compression fails with exception, FileCompressed will print errors in endless loop without any progress:

2018-04-26 11:35:25.152 [ERROR][wal-file-compressor%DPL_GRID%DplGridNodeName][o.a.i.i.p.c.p.w.FileWriteAheadLogManager] Unexpected error during WAL compression
java.io.EOFException: EOF at position [0] expected to read [29] bytes
  at org.apache.ignite.internal.processors.cache.persistence.wal.FileInput.ensure(FileInput.java:126)
  at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.readSerializerVersionAndCompactedFlag(FileWriteAheadLogManager.java:2144)
   at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileCompressor.compressSegmentToFile(FileWriteAheadLogManager.java:1917)
  at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileCompressor.run(FileWriteAheadLogManager.java:1884)
2018-04-26 11:35:25.391 [INFO ][node-stopper][o.a.i.i.m.d.GridDeploymentLocalStore] Removed undeployed class: GridDeployment [ts=1524730971870, depMode=SHARED, clsLdr=union-module-impl:com.sbt.core.envelope.container.loader.ImplClassLoader@7ee81f01, clsLdrId=7114b010361-b9172884-ab39-4b1c-94ed-6208828f27fe, userVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.GridCacheProcessor$RemovedItemsCleanupTask$1, pendingUndeploy=false, undeployed=true, usage=0]
2018-04-26 11:35:25.400 [INFO ][node-stopper][o.a.i.i.IgniteKernal%DPL_GRID%DplGridNodeName]

We should softly handle this situation: print message in log and continue the compression with next segment.
We also should handle "skipped" segments and don't delete them in deleteObsoleteRawSegments().

Issue Links

• is related to: IGNITE-8429 Unexpected error during incorrect WAL segment decompression, causes node termination.
• links to: GitHub Pull Request #3927
• links to: GitHub Pull Request #3927