[SPARK-21795] Broadcast hint ignored when dataframe is cached

• Type: Question
• Status: Resolved
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 2.2.0
• Component/s: Documentation, SQL
• Assignee: Unassigned
• Reporter: Lior Chaga
• Created: 21/Aug/17 05:39
• Updated: 07/Mar/18 02:34
• Resolved: 07/Mar/18 02:34

Description

Not sure if it's a bug or by design, but if a DF is cached, the broadcast hint is ignored, and spark uses SortMergeJoin.

val largeDf = ...
val smalDf = ...
smallDf = smallDf.cache

largeDf.join(broadcast(smallDf))

It make sense there's no need to use cache when using broadcast join, however, I wonder if it's the correct behavior for spark to ignore the broadcast hint just because the DF is cached. Consider a case when a DF should be cached for several queries, and on different queries it should be broadcasted.

If this is the correct behavior, at least it's worth documenting that cached DF cannot be broadcasted.

Issue Links

• duplicates: SPARK-17859 persist should not impede with spark's ability to perform a broadcast join.