If you want to eventually support random-IO, then a block size of 4096 is too large for the CRC, as it will cause a read-modify-write cycle on the entire 4K. 512-bytes reduces this overhead.
With CRC hardware accelerated now, this is not a big overhead. Without hardware acceleration it is ~800MB/sec for 4096 byte chunks, or 200,000 blocks per second or 25% of one CPU load at 200MB/sec writes. With Hardware acceleration this drops by a factor of 4 to 8.
This is besides the point, a paranoid user could configure smaller CRC chunks and test that. I'm suggesting that 4096 is a much saner default.
Secondly, the disk manufacturers guarantee only a 512-byte atomicity on disk. Linux doing a 4K block write guarantees almost nothing wrt atomicity of that 4K write to disk. On a crash, unless you are running some sort of RAID or data-journal, there is a likelihood of the 4K block that's in-flight getting corrupted.
Actually, disk manufacturers are all using 4096 byte atomicity these days (starting with 500GB platters for most manufacturers) **. HDFS should not target protecting power_of_two_butes data with a checksum, but rather (power_of_two_bytes - checksum_size) data so that the hardware atomicity (and OS page cache) lines up exactly with the hdfs checksum chunk + inlined CRC.
2. An append happens a few days later to extend the file from 9K to 11K. CRC3 is now recomputed for the 3K-sized region spanning offsets 8K-11K and written out as CRC3-new. But there is a crash, and the entire 3K is not all written out cleanly
This can be avoided entirely.
A. The OS and Hardware can avoid partial page writes. ext4 and others can avoid partial page writes. The OS only flushes a page at a time. Hardware these days writes blocks in atomic 4096 byte chunks.
B. The inlined CRC can be done so that a single 4096 byte page in the OS contains all of the data and the crc in an atomic chunk, and the CRC and its corresponding data are therefore not split across pages.
Under the above conditions, the performance would be excellent, and the data safety higher than the current situation or any application level crc (unless the application is inlining the crc to prevent splitting the data and crc across pages).
About the transition to 4096 byte blocks on Hard drives ("Advanced Format" disks):