Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-8429

Unexpected error during incorrect WAL segment decompression, causes node termination.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 2.5
    • Fix Version/s: 2.5
    • Component/s: persistence
    • Labels:

      Description

      File decompressor failure due to incorrect (zero-length) archived segment.

      2018-04-30 00:00:02.811 [ERROR][wal-file-decompressor%DPL_GRID%DplGridNodeName][org.apache.ignite.Ignite] Critical system error detected. Will be handled accordingly to configured handler [hnd=class o.a.i.failure.StopNodeOrHaltFailureHandler, failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.IllegalStateException: Thread wal-file-decompressor%DPL_GRID%DplGridNodeName is terminated unexpectedly]]
      java.lang.IllegalStateException: Thread wal-file-decompressor%DPL_GRID%DplGridNodeName is terminated unexpectedly
      at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDecompressor.run(FileWriteAheadLogManager.java:2104)
      2018-04-30 00:00:02.812 [ERROR][wal-file-decompressor%DPL_GRID%DplGridNodeName][org.apache.ignite.Ignite] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=java.lang.IllegalStateException: Thread wal-file-decompressor%DPL_GRID%DplGridNodeName is terminated unexpectedly]]

      touch 0000000000000754.wal
      zip 0000000000000754.wal.zip 0000000000000754.wal
      ls -l
      rw-rw-r- 1 dmitriy dmitriy 0 май 1 16:40 0000000000000754.wal
      rw-rw-r- 1 dmitriy dmitriy 190 май 1 16:46 0000000000000754.wal.zip

      Archive: /tmp/temp/0000000000000754.wal.zip
      Length Method Size Cmpr Date Time CRC-32 Name
      -------- ------ ------- ---- ---------- ----- -------- ----
      0 Stored 0 0% 2018-05-01 16:40 00000000 0000000000000754.wal
      -------- ------- — -------
      0 0 0% 1 file

      We should softly handle this situation: print message in log and continue the decompression with next segment.
      We also should handle "skipped" segments and don't delete them in deleteObsoleteRawSegments().

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                ivan.glukos Ivan Rakov
                Reporter:
                ivandasch Ivan Daschinskiy
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 10m
                  10m