[SAMZA-952] Add validation for broadcast inputs configured for partitions that don't exist

• Type: Bug
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.10.0
• Assignee: Branislav Cogic
• Reporter: Jake Maes
• Created: 24/May/16 21:39
• Updated: 23/Sep/16 10:25

Description

We had an issue at LinkedIn where the user configured a broadcast input as:

task.broadcast.inputs=kafka-queueing.ads_glob#[0-9]

But the ads_glob topic only has 8 partitions.

This caused the following exception in the container log:

2016-05-24 19:59:20 SamzaContainer [ERROR] Caught exception in process loop.
org.apache.samza.SamzaException: No metadata available for partition null.
	at org.apache.samza.checkpoint.OffsetManager$$anonfun$loadDefaults$1$$anonfun$apply$15.apply(OffsetManager.scala:412)
	at org.apache.samza.checkpoint.OffsetManager$$anonfun$loadDefaults$1$$anonfun$apply$15.apply(OffsetManager.scala:380)
	at scala.collection.mutable.HashSet.foreach(HashSet.scala:79)
	at org.apache.samza.checkpoint.OffsetManager$$anonfun$loadDefaults$1.apply(OffsetManager.scala:380)
	at org.apache.samza.checkpoint.OffsetManager$$anonfun$loadDefaults$1.apply(OffsetManager.scala:378)
	at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
	at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
	at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
	at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
	at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
	at org.apache.samza.checkpoint.OffsetManager.loadDefaults(OffsetManager.scala:378)
	at org.apache.samza.checkpoint.OffsetManager.start(OffsetManager.scala:175)
	at org.apache.samza.container.SamzaContainer.startOffsetManager(SamzaContainer.scala:604)
	at org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:550)
	at org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:93)
	at org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:67)
	at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)

We should add some validation and a meaningful error message for cases when the broadcast inputs don't make sense compared to the metadata.