[SPARK-13121] java mapWithState mishandles scala Option

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 1.6.1, 2.0.0
• Component/s: DStreams, Java API
• Assignee: Gabriele Nizzoli
• Reporter: Gabriele Nizzoli
• Created: 01/Feb/16 23:20
• Updated: 02/Feb/16 21:23
• Resolved: 02/Feb/16 21:20

Description

in Spark Streaming, java mapWithState that uses Function3 has a bug in the convertion from a scala Option to a java Optional. In the conversion, the code in `StateSpec.scala`, line 222 is
`Optional.fromNullable(v.get)`. This fails if `v`, an `Option`, is `None`, better to use `JavaUtils.optionToOptional(v)` instead.

Workaround is to use the Function4 call to mapWithState. This call has the right conversion.

Issue Links

• links to: [Github] Pull Request #11007 (gabrielenizzoli)
• links to: [Github] Pull Request #11028 (gabrielenizzoli)