[BEAM-12529] to_pcollection does not support timestamp types

• Type: Improvement
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.32.0
• Component/s: dsl-dataframe
• Assignee: Brian Hulette
• Reporter: Brian Hulette
• Created: 22/Jun/21 21:27
• Updated: 29/Jun/21 17:19
• Resolved: 29/Jun/21 17:19

Description

We do not currently support converting timestamps directly from a deferred dataframe to a schema'd PCollection. It will raise an error like:

TypeError: Unsupported dtype in proxy: 'datetime64[ns]'

Note that pandas timestamps are nanosecond precision by default, while Beam's Timestamp type is microsecond precision.

Issue Links

• links to: GitHub Pull Request #15073