I think the probability gets larger the more disks/server, and now that 12HDD units are coming out, you can plan to see it some time after you spec out your next datacentre.
- deletion of large block size files can leave a disk unbalanced.
- MR temp space in the same disks can fill up then free disks
- Replacement of failed HDDs leaves that disk permanently underutilised.
the third one is new; on a 12 disk server, with most of all 12 disks allocated to HDFS, one block in 12 would go to any specific disk. If one disk is replaced, it still only gets 1/12 of the blocks, even though if all the other disks were 70-80% full, its the disk with the most space. The disks would only be balanced if the new disk got more of the writes (which could have adverse consequences for future IO rates), or some rebalancing on a single machine moves data from one disk to another (or to be precise, copies, validates the block checksums, then deletes).
I actually think HDFS-1121 should come first: provide a way of measuring the distribution on disks on a single server. Once we have the data we can start worrying about ways to correct any distribution issues.