[KAFKA-9372] Add producer config to make topicExpiry configurable

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 1.1.0
• Component/s: producer
• Assignee: Brian Byrne
• Reporter: Jiao Zhang
• Created: 07/Jan/20 02:42
• Updated: 27/Feb/20 07:15
• Resolved: 27/Feb/20 07:15

Description

Sometimes we got error "org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 1000 ms" on producer side. We did the investigation and found

1. our producer produced messages in really low rate, the interval is more than 10 minutes
2. by default, producer would expire topics after TOPIC_EXPIRY_MS, after topic expired if no data produce before next metadata update (automatically triggered by metadata.max.age.ms) partitions entry for the topic would disappear from the Metadata cache As a result, almost for every time's produce, producer need fetch metadata which could possibly end with timeout.

To solve this, we propose to add a new config metadata.topic.expiry for producer to make topicExpiry configurable. Topic expiry is good only when producer is long-lived and is used for producing variable counts of topics. But in the case that producers are bounded to single or few fixed topics, there is no need to expire topics at all.