Major Description
During longevity testing of the encryption-at-rest version of Accumulo, we would occasionally see IOExceptions that took the form of Zlib throwing an "incorrect header check" exception. These exceptions occurred only after a few hours of testing, during minor and major compaction of various RFiles. Downloading and examining the RFiles in question showed no obvious deformities within the RFile structure.
Some careful debugging later, the crux of the problem turned out to be some calls to read() when readFully() should have been used.
Patch coming forthwith. Also included in this patch is another secret key handling strategy that caches the secret key from HDFS when first read.