Details
-
Technical Debt
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
Description
Some FileInputFormats don't use FileInputFormat#createSplits (that would detect that the file is non-splittable and deal with reading boundaries correctly), they all create split manually from FileSourceSplit. If input files are compressed, split length is determined by the compressed file length leading to this bug. We should force reading the whole file split (like it is done for binary input formats) on compressed files. Parallelism is still done at the file level (as now)
Attachments
Issue Links
- links to