[BEAM-8495] Change byte size estimate for compressed files for FileBasedSources

• Type: Bug
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: io-java-files, io-py-files
• Assignee: Unassigned
• Reporter: Ahmet Altay
• Created: 26/Oct/19 05:06
• Updated: 04/Jun/22 13:51

Description

Consider change the estimated byte size for file based sources for better initial autoscaling (on runners which support it; currently only Dataflow). Currently the estimate is based on total size, for compressed file we could add a factor multiplier (~5) for compressed files in a blob go produce a better estimate.