[BEAM-9496] Add a Dataframe API for Python

• Type: New Feature
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Fix Version/s: 2.32.0
• Component/s: dsl-dataframe, sdk-py-core
• Labels: dataframe-api
• Assignee: Unassigned
• Reporter: Robert Bradshaw
• Created: 13/Mar/20 00:08
• Updated: 16/Sep/21 19:16
• Resolved: 16/Sep/21 19:16

Description

This is an umbrella bug for the dataframes work.

Issue Links

• is a parent of: BEAM-11505 pandas IOs should set chunksize by peeking at input data at construction time
• is a parent of: BEAM-9547 Implement all pandas operations (or raise WontImplementError)
• is a parent of: BEAM-9545 MVP: DataframeTransform
• is a parent of: BEAM-9546 Support for batching a schema-aware PCollection and processing as a Dataframe
• is a parent of: BEAM-10619 Report ratio of implemented pandas tests
• is a parent of: BEAM-10720 Implement pandas StringMethods
• is a parent of: BEAM-11052 to_pcollection should memoize on DF expressions
• is a parent of: BEAM-11163 Implement DataFrame.rolling
• is a parent of: BEAM-9561 Run pandas tests with Beam Dataframe API
• is duplicated by: BEAM-10035 Pandas Dataframes API
• is duplicated by: BEAM-9544 Add Pandas-compatible Dataframe API
• is related to: BEAM-10626 tests fails on windows - pandas_doctests_test.DoctestTest
• is related to: BEAM-11064 Consider adding a pandas typehint
• links to: GitHub Pull Request #10757
• links to: GitHub Pull Request #11264

Sub-Tasks

• Stronger testing of dataframes partitioning declartions.: Resolved
• Intelligently partition dataframes: Resolved
• Report global ratio of implemented pandas tests: Resolved