[KAFKA-14011] Reduce retention.ms from default 7 days to 1 day (Make it configurable) for the dead letter topic created for error handling via sink connector

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.1
• Component/s: config, connect

Language

English Java

• Assignee: Unassigned
• Reporter: Souvik
• Created: 20/Jun/22 07:55
• Updated: 20/Jun/22 11:32

Description

We are creating a sink connector along with the error handling mechanism , so if in case if there is bad record it is routed to error queue , The properties used while creating the sink connector are as following

 
'errors.tolerance'='all',
'errors.deadletterqueue.topic.name' = 'error_dlq',
'errors.deadletterqueue.topic.replication.factor'= -1,
'errors.log.include.messages' = true,
'errors.log.enable' = true,
'errors.deadletterqueue.context.headers.enable' = true
now is there any way we can configure the retention.ms for the dead letter queue topic i.e from default 7 days to 1 day , or is there any way we can change the compaction policy by providing any property while creating the source connector
we have tried the following properties in the connector

'topic.creation.default.retention.ms'='3000',
'admin.topic.creation.default.retention.ms'='3000',
'admin.retention.ms' = '3000',
'admin.topic.retention.ms' = '3000',
'admin.topic.creation.retention.ms' = '3000',
'error.topic.creation.default.retention.ms'='3000',
'error.deadletterqueue.topic.creation.default.retention.ms'='3000',
'error.deadletterqueue.topic.retention.ms'='3000',
'admin.topic.creation.default.retention.ms' = 3000, 

but it did not work.