I recently remarked that if there are duplicated elements in the argument of write.partitionBy(), then the same partition subdirectory will be created multiple times.
The above code will produce an output directory with this structure:
And we won't be able to read the output
I am not sure if partitioning a dataframe twice by the same column make sense in some real-world applications, but it will cause schema inference problems in tools like AWS Glue crawler.
Should Spark handle the deduplication of the partition columns? Or maybe throw an exception when duplicated columns are detected?
If this behaviour is unexpected, I will work on a fix.