[KAFKA-7653] Streams-Scala: Add type level differentiation for Key and Value serdes.

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: streams
• Labels: kip scala
• Assignee: Unassigned
• Reporter: Mark Tranter
• Created: 18/Nov/18 08:40
• Updated: 29/Dec/22 00:35

Description

Implicit resolution/conversion of Serdes/Consumed etc is a big improvement for the Scala Streams API. However in cases where a user needs to differentiate between Key and Value serializer functionality (i.e. using the Schema Registry), implicit resolution doesn't help and could cause issues. 

e.g.

case class MouseClickEvent(pageId: Long, userId: String)

builder
  // Long serde taken from implicit scope configured with
  // `isKey` = true
  .stream[Long, MouseClickEvent]("mouse-clicks")
  .selectKey((_,v) => v.userId)
  .groupByKey
  .aggregate(() => 0L, (_: String, mce: MouseClickEvent, count: Long) => count + 1)
  .toStream
  // Same Long serde taken from implicit scope configured with
  // `isKey` = true, even thought the `Long` value in this case
  // will be the Value
  .to("mouse-clicks-by-user")

It would be ideal if Key and Value Serde/SerdeWrapper types/type classes could be introduced to overcome this limitation.

KIP-513: https://cwiki.apache.org/confluence/display/KAFKA/KIP-513%3A+Distinguish+between+Key+and+Value+serdes+in+scala+wrapper+library+for+kafka+streams

Issue Links

• links to: GitHub Pull Request #5938