[BEAM-9275] BIP-1: Beam Schema Options

• Type: Improvement
• Status: In Progress
• Priority: P3
• Resolution: Unresolved
• Component/s: sdk-java-core
• Labels: Clarified
• Assignee: Unassigned
• Reporter: Alex Van Boxel
• Created: 09/Feb/20 22:14
• Updated: 04/Jun/22 14:45

Description

Introduce the concept of Options in Beam Schema’s to add extra context to fields and schemas. In contrast to the current Beam metadata that is present in a FieldType, options would be added to fields, logical types and schemas. The schema convertors (ex. Avro, Proto, …) can add options/annotations/decorators that were in the original schema to the Beam schema with these options. These options, that add contextual metadata, can be used in the pipeline for specific transformations or augment the end schema in the target output.

Issue Links

• mentioned in: Page Loading...

Sub-Tasks

• BIP-1: Typed options for Row Schema and Fields: Triage Needed
• BIP-1: Convert protobuf options to Schema options: Triage Needed
• BIP-1: Convert avro metadata to Schema options: Resolved
• BIP-1: Remove schema metadata usage for Protobuf extension: Resolved
• BIP-1: Rename setRowOption to setOption on Option builder: Resolved
• BIP-1: Deprecate and remove FieldType metadata: Triage Needed