[BEAM-2274] beam on spark runner run much slower than using spark

• Type: Test
• Status: Resolved
• Priority: P2
• Resolution: Duplicate
• Fix Version/s: Not applicable
• Component/s: runner-spark
• Assignee: Unassigned
• Reporter: liyuntian
• Created: 12/May/17 00:13
• Updated: 16/May/20 14:05
• Resolved: 05/Mar/20 15:16

Description

I run a job,read hdfs files using Read.from(HDFSFileSource.from()) and do some ParDo.of functions. and I also run the same job, read hdfs file using sc.textFile(file) and do some RDDs.but I find beam job is much slower than spark job.Is there something that beam should improve or something wrong with my system and my code?thank you.

Issue Links

• is superceded by: BEAM-9440 Performance Issues with Beam Runners compared with Native Systems
• relates to: BEAM-9440 Performance Issues with Beam Runners compared with Native Systems