[SPARK-30732] BroadcastExchangeExec does not fully honor "spark.broadcast.compress"

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.4.0
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Puneet
• Created: 05/Feb/20 00:51
• Updated: 10/Feb/20 03:19
• Resolved: 10/Feb/20 03:19

Description

Setting spark.broadcast.compress to false disables compression while sending broadcast variable to executors (https://spark.apache.org/docs/latest/configuration.html#compression-and-serialization)

However this does not disable compression for any child relations sent by the executors to the driver.

Setting spark.boradcast.compress to false should disable both sides of the traffic, allowing users to disable compression for the whole broadcast join for example.

https://github.com/puneetguptanitj/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/BroadcastExchangeExec.scala#L89

^here `executeCollectIterator` calls `getByteArrayRdd` which by default always gets a compressed stream