[KAFKA-16310] ListOffsets doesn't report the offset with maxTimestamp anymore

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.7.0
• Fix Version/s: 3.8.0
• Assignee: Chia-Ping Tsai
• Reporter: Emanuele Sabellico
• Created: 28/Feb/24 10:55
• Updated: 3 days ago 01:05
• Resolved: 10/Apr/24 03:38

Description

Updated: This is confirmed a regression issue in v3.7.0.

The impact of this issue is that when there is a batch containing records with timestamp not in order, the offset of the timestamp will be wrong.(ex: the timestamp for t0 should be mapping to offset 10, but will get offset 12.. etc). It'll cause the time index is putting the wrong offset, so the result will be unexpected.

===
The last offset is reported instead.
A test in librdkafka (0081/do_test_ListOffsets) is failing an it's checking that the offset with the max timestamp is the middle one and not the last one. The tests is passing with 3.6.0 and previous versions

This is the test:
https://github.com/confluentinc/librdkafka/blob/a6d85bdbc1023b1a5477b8befe516242c3e182f6/tests/0081-admin.c#L4989

 

there are three messages, with timestamps:

t0 + 100
t0 + 400
t0 + 250

and indices 0,1,2. 

then a ListOffsets with RD_KAFKA_OFFSET_SPEC_MAX_TIMESTAMP is done.
it should return offset 1 but in 3.7.0 and trunk is returning offset 2

Even after 5 seconds from producing it's still returning 2 as the offset with max timestamp.
ProduceRequest and ListOffsets were sent to the same broker (2), the leader didn't change.

%7|1709134230.019|SEND|0081_admin#producer-3| [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ProduceRequest (v7, 206 bytes @ 0, CorrId 2) %7|1709134230.020|RECV|0081_admin#producer-3| [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received ProduceResponse (v7, 95 bytes, CorrId 2, rtt 1.18ms) %7|1709134230.020|MSGSET|0081_admin#producer-3| [thrd:localhost:39951/bootstrap]: localhost:39951/2: rdkafkatest_rnd22e8d8ec45b53f98_do_test_ListOffsets [0]: MessageSet with 3 message(s) (MsgId 0, BaseSeq -1) delivered 

%7|1709134235.021|SEND|0081_admin#producer-2| [thrd:localhost:39951/bootstrap]: localhost:39951/2: Sent ListOffsetsRequest (v7, 103 bytes @ 0, CorrId 7) %7|1709134235.022|RECV|0081_admin#producer-2| [thrd:localhost:39951/bootstrap]: localhost:39951/2: Received ListOffsetsResponse (v7, 88 bytes, CorrId 7, rtt 0.54ms)

Issue Links

• links to: GitHub Pull Request #15461
• links to: GitHub Pull Request #15618
• links to: GitHub Pull Request #15621

Sub-Tasks

• Fix un-compressed records: Resolved
• Fix compressed records: Resolved