[SPARK-30023] Spark partitionby saves as columnName={value} | Can it be only columnvalue

• Type: Question
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.4.3
• Component/s: Spark Core, SQL
• Labels: dataframe
• Assignee: Unassigned
• Reporter: ShivaKumar SS
• Created: 25/Nov/19 09:20
• Updated: 12/Dec/22 18:11
• Resolved: 02/Dec/19 02:39

Description

I am using scala and spark.

This is using Dataframe and in dataframe i have a columns by name "year" "month" and "date" and many other columns which are not relevant here.

 

Code snippet.
 
{{df.write.partitionBy("year", "month", "date").format("csv").option("header", "true").save(outPath)
}}
 

and my expectation is to save in a hierarchy folder structure.
 
2016/11/15/file*.csv

 

but the files are getting saved as 

 
 
year=2016/month=11/date=15/file*.csv

 

Is there any way i can remove the column name from the directory structure and save only the column value here. ?