[BEAM-11869] Profile and improve performance for the local FnApiRunner

• Type: Sub-task
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: runner-py-direct, sdk-py-core
• Labels: gsoc gsoc2021 mentor
• Assignee: Unassigned
• Reporter: Pablo Estrada
• Created: 25/Feb/21 19:34
• Updated: 11/May/21 17:19

Description

The FnApiRunner is undergoing a series of changes to support streaming. These changes are altering its execution significantly, and may introduce inefficiencies.

This project has the following deliverables:

• A report with results from profiling the execution of a pipeline, and finding hotspots, and inefficiencies
• Code improvements to speed up the execution of the FnApiRunner
• Improvements to the FnApiRunner manual to instruct others on how to do profiling.

 

Tools that you may need to use:

• Benchmarks for the Python direct runner in http://metrics.beam.apache.org
• The FnApiRunner and TestStream microbenchmarks in sdks/python/apache_beam/tools/
• A profiling tool like: https://pypi.org/project/flameprof/ or https://github.com/uber-archive/pyflame 
• Google Docs

Contact  Pablo in [e-mail] to ask questions about this project.