Details
-
Documentation
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
2.4.4
-
None
Description
Hiļ¼the description of `spark.sql.files.maxPartitionBytes` is shown as below.
The maximum number of bytes to pack into a single partition when reading files.
It seems that it can ensure each partition at most process bytes of that value for spark sql.
As shown in the attachment, the value of spark.sql.files.maxPartitionBytes is 128MB.
For stage 1, its input is 16.3TB, but there are only 6400 tasks.
I checked the code, it is only effective for data source table.
So, its description is confused.
Same as all the descriptions of `spark.sql.files.*`.
