[BEAM-3437] Support schema in PCollections

• Type: Wish
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.5.0
• Component/s: beam-model
• Assignee: Reuven Lax
• Reporter: Jean-Baptiste Onofré
• Created: 09/Jan/18 10:17
• Updated: 16/May/20 13:51
• Resolved: 09/Jan/19 10:55

Description

As discussed with some people in the team, it would be great to add schema support in PCollections. It will allow us:

1. To expect some data type in PTransforms
2. Improve some runners with additional features (I'm thinking about Spark runner with data frames for instance).

A technical draft document has been created:

https://docs.google.com/document/d/1tnG2DPHZYbsomvihIpXruUmQ12pHGK0QIvXS1FOTgRc/edit?disco=AAAABhykQIs&ts=5a203b46&usp=comment_email_document

I also started a PoC on a branch, I will update this Jira with a "discussion" PR.

Issue Links

• links to: GitHub Pull Request #4964