Details
-
Improvement
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
2.0.0
-
None
-
None
Description
Right now, we hold the meta store full snapshot in memory until it finishes. Where there are a lot of tables, the size could be really big and may cause out of memory issue.
It is better to stream the snapshot processing, saving into sentry data store while getting next table. In this way, the memory requirement for getting full snapshot can drop dramatically.
When we finish, we save the notificationID right before taking the snapshot. Then get notifications with ID larger than notificationIDBefore, and process them as normal notification events.
We don't need atomic write of the snapshot in a single transaction. We may be able to do it in simpler ways