When calling FileUtils.checksumCRC32 from multiple threads (in order to improve throughput when calculating CRC's for a large folder), the code is not thread-safe, resulting in incorrect CRC output.
The following simple test demonstrates the issue:
In the above code, with a thread-pool size of 1, all calculated CRC's for the file are the same. With a thread-pool size of more, the CRC's differ.
The issue appears to be related to the use of a common SKIP_BYTE_BUFFER in IOUtils.consume. The multiple threads all read into the same buffer as the data is being "discarded". However, FileUtils.checksum uses a CheckedInputStream to calculate the CRC, which uses the value read into the shared buffer. With multiple threads writing to that buffer the CRC mechanism breaks down.