[HDFS-2535] A Model for Data Durability

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: documentation
• Assignee: Robert Chansler
• Reporter: Robert Chansler
• Created: 03/Nov/11 17:23
• Updated: 20/Nov/17 06:34
• Resolved: 03/Nov/11 17:26

Description

This is a statistical model considering data durability in HDFS in the presence of Data Node failures. The attached spreadsheet considers the probability of losing a block with three replicas in the case of uncorrelated failures of DNs. Also included is a section that looks at the consequences of simultaneous failures.

The model parameters reflect experience at Yahoo with a large cluster. But it is easy to change the parameters in the spreadsheet. Number of replicas is not a easily adjusted parameter. And while published reports (the Google papers) suggest that node failures are not really uncorrelated, this does give some practical insight into HDFS durability.

I and others have quoted from this work in the past. I thought it good to make the details conveniently available.