[FLUME-2848] Kafka Source could not work with Kafka Sink

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.6.0
• Component/s: Sinks+Sources
• Labels: easyfix

Environment

IMHO any

• Assignee: Unassigned
• Reporter: Arkadiusz Firus
• Created: 01/Dec/15 09:44
• Updated: 19/May/16 17:06

Description

KafkaSource stores information from which topic a record was read. This information is stored in an event header with the key "topic" (see org.apache.flume.source.kafka.KafkaSource line 108).
KafkaSink can be instructed to store record in a different topic than the one in configuration (dynamic destination). To do that "topic" header have to be set in an event (see org.apache.flume.sink.kafka.KafkaSink line 107).

When this two features are combined we have infinite loop - KafkaSink writes records to the topic which is read by KafkaSource.

Two solve the problem fields:
org.apache.flume.sink.kafka.KafkaSink.TOPIC_HDR
org.apache.flume.source.kafka.KafkaSourceConstants.TOPIC
need to have different values.

Issue Links

• relates to: FLUME-2789 Kafka sink sends messages to the source topic by mistake