1. test is doing a short circuited read of a 13 byte file doTestShortCircuitRead(false, size= 13, readOffset= 0)
2. # creates 13 bytes of data, saves it: byte fileData = AppendTestUtil.randomBytes(seed, size)
3. dest buffer is created for this
byte actual = new byte[expected.length-readOffset];
4 does some small reads:
int nread = stm.read(actual, 0, 3);
nread += stm.read(actual, nread, 2);
nread += stm.read(actual, nread, 517);
The exception is being raised as the code is asking to read 517 bytes into a buffer 13 bytes long. This breaks IOStream's rules: you can't ask for more than you have space for. It says that clearly in the IOStream API spec; what was added in
HADOOP-12994 was the checking of passing in too big a length or negative offsets.
I think this a bug in the test. Whatever it is trying to do, it shouldn't be trying to do it on such a small buffer.
What's interesting though is when you delve into the code: the block reader logic doesn't look at the length of the read at all. That is, it appears to fill up the entire byte array passed in, from the offset supplied, stopping at the end of the buffer or file, whichever comes first.
Which is something that other code (i.e. production code) could be relying on. They shouldn't, as the code will break when working with any FS other than HDFS, but there is a risk that they might.
What to do?
- retain checks, fix test.
- log at warning and shrink len parameter when passed down. People shouldn't be doing this, but HDFS will reluctantly let you.