Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
If you consider Kafka to be a "database" with "transaction logs", you need a way for backup/recovery just like databases do.
The beauty of such a solution would be to enable Kafka for smaller scenarios where you do not want to have a large cluster. You could even use a single node Kafka. In worst case you lose all data since the backup and you have to ask the sources to send that data again - for most that is possible.
Currently you have multiple options, none of which are good.
- Setup Kafka fault tolerant and with replication factors: Needs a larger server and does not prevent many types of problems like software bugs, deleting a topic by accident,...
- Mirror Kafka: Very expensive.
- Shutdown Kafka, disk copy, startup Kafka
- Add a database before Kafka as a primary persistence: Very very expensive and forfeits the idea of Kafka
I wonder what really is needed for an online backup strategy. If I am not mistaken it is relatively little.
- A command that causes Kafka to switch to new files so that the file containing all past data do not change any longer.
- Export of the current Zookeeper values, unless they can be recreated from the transaction log files anyhow.
- Then you can backup the Kafka files
- A command that tells that the backup is finished to cleanup things.
- Later a way to merge the recovered backup instance with the Kafka log written since then up to a certain point. Example: Backup was taken at midnight, delete topic was done a 11:00. You start with the backup, apply the logs until 10:59 and then you bring up Kafka fully online again.