[BEAM-14438] Beam slowness compared to flink-native

• Type: Bug
• Status: Triage Needed
• Priority: P2
• Resolution: Unresolved
• Affects Version/s: 2.38.0
• Component/s: runner-flink
• Labels: performance
• Assignee: Unassigned
• Reporter: Ifat Afek
• Created: 09/May/22 08:28
• Updated: 05/Jun/22 00:22

Description

I tried to compare a very simple beam pipeline with an equivalent flink-native pipeline. Both pipelines read strings from one kafka topic and write them to another topic. I ran each pipeline separately on a single task manager with a single slot and parallelism 1.

Flink native runs 5 times faster than beam: 150,000 strings per second in flink comparing to 30,000 in beam.

When using Avro and schema registry the difference is even more significant - flink native runs 30-80 times faster than beam.

Attached is the java code of both string-to-string pipelines.