[KYLIN-4964] Receiver consumer thread should be stoped while encounting unrecoverable error

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: v3.1.1
• Fix Version/s: v3.1.3
• Component/s: Real-time Streaming
• Assignee: Xingxing Di
• Reporter: Xingxing Di
• Created: 08/Apr/21 13:32
• Updated: 06/Jan/22 06:52
• Resolved: 09/Nov/21 09:21

Description

org.apache.kylin.stream.core.storage.columnar.ColumnarMemoryStorePersister#persist will catch exception and just log it, this behavior will cause critical problem, like our case: there is no space left on device, `persist` method failed for many times, we lost several hours of data.

Here is our solution which already tested:

1. Throw IllegalStorageException while we cannot do the persist
2. Then stop the consumer thread
3. Add `consumer_thread_alive` in ConsumerStats for monitoring
4. Also fix an another issue which will cause wrong result after restore from checkpoint