[BEAM-1026] User guide - "How to create Beam IO Transforms"

• Type: Task
• Status: Resolved
• Priority: P2
• Resolution: Duplicate
• Fix Version/s: Not applicable
• Component/s: website
• Assignee: Unassigned
• Reporter: Stephen Sisk
• Created: 21/Nov/16 21:46
• Updated: 16/May/20 13:55
• Resolved: 22/Nov/16 01:28

Description

Beam has javadocs for how to create a read or write transform, but no friendly user guide on how to get started using BoundedSource/BoundedReader.

This should cover:

• background on beam's source/sink API design
• design patterns
• evaluating different data sources (eg, what are the properties of a pub sub system that affect how you should write your UnboundedSource? What is the best design for reading from a NoSql style source?)
• testing - how to write unit, integration (and once we have them, performance tests)
• public API recommendations

This is related, but not strictly overlapping with:
https://issues.apache.org/jira/browse/BEAM-193

• the Dataflow SDK documentation for "Custom Sources and Sinks" contains some info about writing Sources/Sinks, but it is somewhat out of date, and doesn't reflect the things we've learned recently.

Issue Links

• is a clone of: BEAM-1025 User guide - "How to create Beam IO Transforms"