[SAMZA-1167] New streamId-specific configs do not override equivalent system-scoped configs

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.13.0
• Assignee: Jake Maes
• Reporter: Jake Maes
• Created: 24/Mar/17 18:14
• Updated: 29/Mar/17 20:59
• Resolved: 29/Mar/17 20:59

Description

Thanks to [e-mail] for finding this.

Test Case:
With the following config, the json key serde is not used

job.factory.class=org.apache.samza.job.local.ThreadJobFactory
job.name=wikipedia-parser
app.class=org.apache.samza.test.integration.TestSessionizerApplication
serializers.registry.json.class=org.apache.samza.serializers.JsonSerdeFactory
serializers.registry.string.class=org.apache.samza.serializers.StringSerdeFactory

streams.page-views.samza.key.serde=json
streams.page-views.samza.msg.serde=string
streams.page-views.samza.system=kafka

systems.kafka.samza.factory=org.apache.samza.system.kafka.KafkaSystemFactory
systems.kafka.consumer.zookeeper.connect=localhost:54567
systems.kafka.consumer.auto.offset.reset=largest
systems.kafka.producer.bootstrap.servers=localhost:54570
systems.kafka.samza.msg.serde=string
systems.kafka.samza.key.serde=string
systems.kafka.samza.offset.default=oldest

job.coordinator.system=kafka
job.coordinator.replication.factor=1
task.window.ms=1000

StreamSpec streamSpec = new StreamSpec("page-views", "PageViewEvent", "kafka");

Bug:
SamzaContainer uses getSerdeStreams() to get a list of the streams that have custom serdes and that method doesn’t pick up the overrides. So, it never asks for the system specific properties.

Issue Links

• links to: GitHub Pull Request #101