[SPARK-34013] Data in redis was obtained using mapPartitions Sparkstreaming, which was not as expected

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.4.2
• Component/s: Spark Core, Structured Streaming
• Assignee: Unassigned
• Reporter: xiyang
• Created: 05/Jan/21 12:23
• Updated: 05/Jan/21 13:14

Description

 

rdd.mapPartitions(par => {
 val jedis: Jedis = AdJedisPoolUtils.getJedis
 jedis.select(4)
 val res: Iterator[(((String, String), (String, String, String, String, String, String, String, String, String)), String)] = par.map(data =>{ 
println(data._2._1) 
val tagId: String = jedis.hget("game_tag_union", data._2._1) 
println(tagId) 
(data, tagId) 
})
jedis.close()
 res
 })

 

The final result was completely different from the one foreachPartition got, but foreachPartition got the correct solution

eg:

rdd.foreachPartition(par=>{
val jedis: Jedis = AdJedisPoolUtils.getJedis
 jedis.select(4)
par.foreach(data=>{ 
val tagId: String = jedis.hget("game_tag_union", data._2._1) println(tagId) 
})
jedis.close()
})