[FLUME-3377] file channel data file corrupted, file channel can't be started

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.9.0
• Component/s: File Channel

Environment

Run in Kubernetes cluster, with 4 replicas, each of which has its own separate persistent volume for file channel, and there were almost 95% free disk space when the issue occured.

• Assignee: Unassigned
• Reporter: Dio Jin
• Created: 30/Jul/20 06:52
• Updated: 30/Jul/20 06:52

Description

Hi, we used flume 1.9.0 to ingest data from Kafka to HDFS, our config file is attached. It ran smoothly for some time, however, it currently failed to ingest data and kept throwing error logs, some important log is attached.  Per log, the file channel failed to be started due to corrupted data file, and it tried relentlessly but always failed.  The flume instance is hosted in Kubernetes and has 4 replicas, each of which has its own separate persistent volume for file channel, and there was almost 95% free disk space when the issue occured.

So there are two questions, 

1. what is the cause for the corrupted data files? since it is our production apps, and we trust flume's robustness, so we don't expect to see this corrupted data file. Moreover, how could we avoid such corrupted data files?
2. How do we resume from this situation without losing any data in channel? Removing checkoutDir and dataDir isn't acceptable.    

Thanks very much.

Here are some very key logs, full logs can be seen in attached file. 

org.apache.flume.channel.file.FileChannel.start(FileChannel.java:295)] Failed to start the file channel [channel=channel2HDFS1]
2020-07-29T07:15:31.640949847Z java.lang.RuntimeException: org.apache.flume.channel.file.CorruptEventException: Could not parse event from data file. 

2020-07-29T07:15:31.638860323Z at org.apache.flume.channel.file.TransactionEventRecord.fromByteArray(TransactionEventRecord.java:212)

...

2020-07-29T07:15:31.64750767Z 2020-07-29 00:15:31,646 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:158)] Unable to deliver event. Exception follows.
2020-07-29T07:15:31.647539686Z java.lang.IllegalStateException: Channel closed [channel=channel2HDFS1]. Due to java.lang.RuntimeException: org.apache.flume.channel.file.CorruptEventException: Could not parse event from data file.
2020-07-29T07:15:31.647552984Z at org.apache.flume.channel.file.FileChannel.createTransaction(FileChannel.java:358)