[HBASE-2070] Collect HLogs and delete them after a period of time

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.90.0
• Hadoop Flags: Reviewed
• Assignee: Jean-Daniel Cryans
• Reporter: Jean-Daniel Cryans
• Created: 23/Dec/09 21:40
• Updated: 20/Nov/15 13:01
• Resolved: 18/Feb/10 00:09

Description

For replication we need to be able to service clusters that are a few hours behind in edits. For example, after distcp'ing a snapshot of the DB to another cluster, we need to make sure we get the edits that came in after the snapshot was taken.

I plan the following changes:

• Instead of deleting HLogs during a log roll or after a log split, move them to another folder where all logs should be aggregated.
• Add a new configuration for how old a log can be. For a normal cluster I think of a default of 2 hours. For replication you may want to set it much higher.
• Create a new thread in the master that checks for logs older than configured time and that deletes them.

I also fancy having the deletion time to be configurable while the cluster is running. I'm also thinking of adding a way to tell the cluster to replay edits on itself.