[BEAM-8576] Regression when reading many files

• Type: Bug
• Status: Resolved
• Priority: P3
• Resolution: Won't Fix
• Affects Version/s: 2.14.0, 2.15.0, 2.16.0
• Fix Version/s: Missing
• Component/s: runner-spark
• Assignee: Unassigned
• Reporter: Stefan De Smit
• Created: 07/Nov/19 13:28
• Updated: 13/Jan/22 21:52
• Resolved: 14/Sep/21 20:16

Description

When reading many files, I used to get many tasks. (beam 2.12)

When I upgrade to beam 2.14, the same code leads to different execution where all files are read by only 1 task.

This happens when not using the Source but the DoFn's (via 'withHintMatchesManyFiles')

final PCollection<GenericRecord> records = pipeline.apply(AvroIO.readGenericRecords(mySchema)
    .from(options.getInputPath() + "/*/*/*/data/file.avro").withHintMatchesManyFiles());
records.apply(Count.globally())