Actually, one more potential source of confusion is that we use OffsetAndMetadata for both offset commits requests and offset fetch responses.
i.e., an OffsetFetchResponse will contain: offset, metadata and this timestamp field. The timestamp field should really be ignored. It is annoying to document such things - i.e., tell users to just ignore the field.
Ideally, I think we should do the following:
- Remove the timestamp from the OffsetAndMetadata class
- Move it to the top-level of the OffsetCommitRequest and rename it to retentionMs
- The broker will compute the absolute time (based off time of receipt) that the offset should be expired
- The above absolute time will continue to be stored in the offsets topic and the cleanup thread can remove those offsets when they are past their TTL.
- OffsetFetchResponse will just return OffsetAndMetadata (no timestamp)
We (linkedin and possibly others) already deployed this to some of our consumers but if we can bump up the protocol version when doing the above and translate requests that come in with the older version I think it should be okay.