This sounds interesting. One thing I don't completely understand about your proposed API is whether you will have multiple (position, length) pairs as inputs. Traditionally, scatter-gather implies being able to read multiple locations at once, like in ''preadv(2)''. However, I only see one position, length argument in your readFullyScatterGather function.
Also, it seems to me that by mmapping at a fixed address, you could create a single contiguous buffer rather than forcing the user to deal with multiple buffers for a single HDFS file.