I don't believe we can ever have multiple clients writing the same sequence file. Exactly one writer per HDFS file, and writer.add() is always called on behalf of exactly one client. So add() can return, unambiguously, the offset at which that client's data will be written.
Yes. Writes are asynchronous. But looking at the length of the file tells you if the write committed or not. The whole point of what I'm proposing is to allow the check for whether the write succeeded to be asynchronous, too. There is no need to keep track of what's in memory. All there is is "data committed to the file, and therefore visible as written", and "data not yet committed, that we'll re-write if a timeout has expired".
I don't think a status check once every five minutes, per agent, would be a problem. That's a small fraction of the number of POSTs that we do – and we retransmit very aggressively when the collector returns a 500 error, there. So this shouldn't make things worse than they already are.
The interface you're describing – where the collector takes responsibility of the data as soon as it returns an HTTP response – is incompatible with your option (1), because we're not willing to wait a whole minute before returning an HTTP response – we'd run out of request handling threads in the server.
I don't know what measurements you're citing. We never implemented option 1. What we've actually implemented is an unreliable version of (1), where the collector takes responsibility – and leaves data in RAM when it responds. The collector, using the code we've written, simply does not flush data before responding. So those benchmarks don't really apply. Likewise, the LocalFSWriter doesn't commit data before responding. The code we actually have returns "OK" before saving the data to disk.
I see there's substantial disagreement here. So I think it makes sense for me to implement and test at scale before submitting a patch for review. If I submit measurements, at scale, demonstrating what I have in mind would that be likely to sway you?