[BEAM-12169] Allow non-deferred column operations on categorical columns

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: dsl-dataframe, sdk-py-core
• Labels: dataframe-api
• Assignee: Unassigned
• Reporter: Brian Hulette
• Created: 14/Apr/21 17:56
• Updated: 04/Jun/22 20:43

Description

There are several operations that we currently disallow because they produce a variable set of columns in the output based on the data (non-deferred-columns). However, for some dtypes (categorical, boolean) we can easily enumerate all the possible values that will be seen at execution time, so we can predict the columns that will be seen.

Note we still can't implement these operations 100% correctly, as pandas will typically only create columns for the values that are observed, while we'd have to create a column for every possible value.

We should allow these operations in these special cases.

Operations in this category:

• DataFrame.unstack, Series.unstack (can work if unstacked level is a categorical or boolean column)
• Series.str.get_dummies
• Series.str.split
• Series.str.rsplit
• DataFrame.pivot
• DataFrame.pivot_table

Issue Links

• is a child of: BEAM-12133 Tracking: DataFrame API future enhancements
• links to: GitHub Pull Request #16615
• links to: GitHub Pull Request #16677
• links to: GitHub Pull Request #16677

Sub-Tasks

• Implement Series.str.get_dummies() for DataFrame API: Resolved
• Implement Series.str.split() and Series.str.rsplit() for DataFrame API: Resolved
• Implement DataFrame.unstack() and Series.unstack() for DataFrame API: Triage Needed
• Implement DataFrame.pivot() for DataFrame API: Triage Needed
• Implement DataFrame.pivot_table() for DataFrame API: Open
• Implement len(GroupBy) and ngroups for DataFrame API: Open
• Documentation for non-deferred-column operations: Open
• Update DataFrame rsplit() api once pandas rsplit() supports regex: Open