[BEAM-14068] RunInference Benchmarking tests

• Type: Sub-task
• Status: Open
• Priority: P2
• Resolution: Unresolved
• Component/s: sdk-py-core
• Assignee: Anand Inguva
• Reporter: Anand Inguva
• Created: 07/Mar/22 21:59
• Updated: 04/Jun/22 23:33

Description

RunInference benchmarks will evaluate performance of Pipelines, which represent common use cases of Beam + Dataflow in Pytorch, sklearn and possibly TFX. These benchmarks would be the integration tests that exercise several software components using Beam, PyTorch, Scikit learn and TensorFlow extended.

we would use the datasets that's available publicly (Eg; Kaggle). 

Size: small / 10 GB / 1 TB etc

The default execution runner would be Dataflow unless specified otherwise.

These tests would be run very less frequently(every release cycle).  

Issue Links

• links to: GitHub Pull Request #17462
• links to: GitHub Pull Request #17697