[SPARK-45855] Unable to set compression codec for Hive CTAS

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.5.0
• Component/s: SQL

Environment

Spark 3.4.0
Stackable.tech release 23.7.0 which runs spark on K8s.

• Assignee: Unassigned
• Reporter: Tim Robertson
• Created: 09/Nov/23 15:43
• Updated: 09/Nov/23 16:20
• Resolved: 09/Nov/23 16:20

Description

Hi,

We've discovered code that worked in Spark 3.3.0 doesn't in 3.4.0. I can't find anything in the release notes to indicate why, so I wonder if this is a bug. Thank you for looking.

Here we're using our own custom codec, but we noticed we can't set gzip either.

  SparkConf conf = spark.sparkContext().conf();
  conf.set("hive.exec.compress.output", "true");
  conf.set("mapred.output.compression.codec", D2Codec.class.getName()); 
  spark.sql("CREATE TABLE b AS SELECT id FROM a");

This will create the table, but it writes uncompressed files, where Spark 3.3.0 would write compressed files. 

Any advice is appreciated and I can help run tests. We run Spark on K8S using the stackable.tech distribution.

Issue Links

• is caused by: SPARK-43504 Mounts the hadoop config map on the executor pod