> but we also need to minimize the amount of buffered, checksummed data. It's thus preferable to keep the outermost, checksummed buffer small ...
Granted, but changing the readChunk() contract to allow multiple continguous chunks to be read at once doesn't impact buffering. The outermost buffer would still be small. However, when an application does a large read, the current API requires that the data be fetched piecemeal into the InputChecker buffer, checksummed and then copied to the application buffer. Allowing multiple contiguous chunks to be read simultaneously, would allow the application buffer to be passed directly to readChunk() and checksummed in place and thus save a copy.
This is a small performance gain, however, a more important reason is that constraining readChunk() to 1 chunk at a time makes it difficult to support preads.
Implementing efficient support for 'sequential preads' i.e. multiple pread invocations hitting contiguous byte ranges is quite tricky. An implementation has to be pretty clever in order to efficiently serve several concurrent threads all calling pread. It probably needs to keep a pool of file descriptors around and have some complicated logic to determine, among other things:
- if, in a pread invocation, the position on any of its available descriptors is close to the position in the call, so that buffered data can be used
- if the position in a call isn't a good match for any of the descriptors, then whether to seek on an existing descriptor or open a new one. Using an existing descriptor might cause buffered data useful for a subsequent call to be thrown away
On the other hand, an implementation thats not trying support 'sequential preads', can be a lot simpler. It also needs to keep a pool of descriptors. But in each call it can pick an arbitrary descriptor, seek, fetch and return the data. It would also do no buffering (except on 1 canonical descriptor which corresponds to the file position that the implementation reports to clients).
It's not clear that the 'sequential pread' access pattern occurs frequently in application contexts. It certainly doesn't in the use cases we've seen so far, they almost all have very little locality of reference across pread invocations. However, given the readChunk() API above, an application level pread would most likely be split into multiple readChunk() calls and require the 'sequential pread' access pattern to be supported. It feels like this access pattern is being imposed on implementations by the readChunk() API and not by application contexts and that is undesirable. If the readChunk() API allowed multiple chunks to be read at once, then an application level pread would likely result in a single call to readChunk() and would allow for a simple implementation.