[FLINK-36039] Support clean historical event handler records in JDBC event handler

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: kubernetes-operator-1.10.0
• Component/s: Autoscaler
• Labels: pull-request-available
• Assignee: RocMarshal
• Reporter: RocMarshal
• Created: 13/Aug/24 02:29
• Updated: 26/Aug/24 03:30
• Resolved: 26/Aug/24 03:30

Description

Currently, the autoscaler generates a large amount of historical data for event handlers. As the system runs for a long time, the volume of historical data will continue to grow. It is necessary to support automatic cleanup of data within a fixed period.

Based on the creation time timestamp, the following approach for cleaning up historical data might be a way:

• Introduce the parameter autoscaler.standalone.jdbc-event-handler.ttl

• • Type: Duration
  • Default value: 90 days
  • Setting it to 0 means disabling the cleanup functionality.
• In the JdbcAutoScalerEventHandler constructor, introduce a scheduled job. Also, add an internal interface method close for AutoScalerEventHandler & JobAutoScaler  to stop and clean up related logic.

• Cleanup logic:

1. 1. Query the messages with create_time less than (currentTime - ttl) and find the maximum maxId in this collection.
   2. Delete 4096 messages at a time from the collection with IDs less than maxId.
   3. Wait 10 ms between each deletion until the cleanup is complete.
   4. Scan and delete expired data daily 

 

Issue Links

• links to: GitHub Pull Request #865