When short-circuit reads are on, HDFS client slows down when checksums are turned off.
With checksums on, the query takes 45.341 seconds and with it turned off, it takes 56.345 seconds. This is slower than the speeds observed when short-circuiting is turned off.
The issue seems to be that FSDataInputStream.readByte() calls are directly transferred to the disk fd when the checksums are turned off.
Even though all the columns are integers, the data being read will be read via DataInputStream which does
To confirm, an strace of the Yarn container shows
To emulate this without the entirety of Hive code, I have written a simpler test app
The jar will read a file in -bs <n> sized buffers. Running it with 1 byte blocks gives similar results to the Hive test run.