Details
-
Sub-task
-
Status: Resolved
-
Critical
-
Resolution: Incomplete
-
None
-
None
Description
When we save a table to a data source table, it is possible that a writer is responsible to write out a larger number of rows, which can make the generated file very large and cause job failed if the underlying storage system has a limit of max file size (e.g. S3's limit is 5GB). We should bound the size of a file generated by a writer and create new writers for the same partition if necessary.