Description
The spark statistics tracking doesn't correctly assess the size of the uploaded files as it only calls getFileStatus on the zero byte objects -not the yet-to-manifest files. Which, given they don't exist yet, isn't easy to do.
HADOOP-17414 will attach the final length as a custom header to the marker object, and implement getXAttr in the S3A FS to probe for it.
BasicWriteStatsTracker can probe for this custom Xattr if the size of the generated file is 0 bytes; if found and parseable use that as the declared length of the output.
Attachments
Issue Links
- is blocked by
-
HADOOP-17414 Magic committer files don't have the count of bytes written collected by spark
-
- Resolved
-
- links to