[SPARK-11691] Allow to specify compression codec in HadoopFsRelation when saving

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Jeff Zhang
• Created: 12/Nov/15 07:39
• Updated: 12/Dec/22 18:11
• Resolved: 09/Mar/16 11:41

Description

Currently, there's no way to specify compression codec when saving data frame to hdfs. It would nice to allow specify compression codec in DataFrameWriter just as we did in RDD api

def saveAsTextFile(path: String, codec: Class[_ <: CompressionCodec]): Unit = withScope {

Issue Links

• is duplicated by: SPARK-13503 Support to specify the (writing) option for compression codec for TEXT
• links to: [Github] Pull Request #9657 (zjffdu)
• links to: [Github] Pull Request #11324 (maropu)

Sub-Tasks

• Support compression in PySpark: Closed