Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
0.12.0
-
None
-
None
-
None
-
Reviewed
Description
FileBasedOutputSizeReader only includes files in the top level output directory. So if files are stored under subdirectories (For eg: MultiStorage), it does not have the bytes written correctly.
0.11 shows the correct number of total bytes written and this is a regression. A quick look at the code shows that the JobStats.addOneOutputStats() in 0.11 also does not recursively iterate and code is same as FileBasedOutputSizeReader. Need to investigate where the correct value comes from in 0.11 and fix it in 0.12.1/0.13.
Attachments
Attachments
Issue Links
- is related to
-
PIG-2924 PigStats should not be assuming all Storage classes to be file-based storage
- Closed