[SPARK-50160] KafkaWriteTask: support timestamp customisation

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.5.3
• Component/s: SQL, Structured Streaming
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Peter Schulz
• Created: 29/Oct/24 14:21
• Updated: Yesterday 05:50

Description

Currently, there is no way to customise the timestamp of a ProducerRecord produced by the KafkaWriteTask. Here at Wikimedia we often use event-time semantics, so it would be helpful if Kafka records produced via spark could resemble that.

The producer already allows that as stated in the Producer.send API docs:

If CreateTime is used by the topic, the timestamp will be the user provided timestamp or the record send time if the user did not specify a timestamp for the record.

So the proposed feature enables users of the spark kafka output to specify the create-time.

I already checked out the code and was able to adapted it to fulfil our need. Since I couldn't find any (closed) ticket concerned with this topic, I assume no such feature has been denied until now.

Issue Links

• links to: GitHub Pull Request #48695