Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-1025

User guide - "How to create Beam IO Transforms"

    Details

    • Type: Task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: website
    • Labels:
      None

      Description

      Beam has javadocs for how to create a read or write transform, but no friendly user guide on how to get started using BoundedSource/BoundedReader.

      This should cover:

      • background on beam's source/sink API design
      • design patterns
      • evaluating different data sources (eg, what are the properties of a pub sub system that affect how you should write your UnboundedSource? What is the best design for reading from a NoSql style source?)
      • testing - how to write unit, integration (and once we have them, performance tests)
      • public API recommendations

      This is related, but not strictly overlapping with:
      https://issues.apache.org/jira/browse/BEAM-193

      • the Dataflow SDK documentation for "Custom Sources and Sinks" contains some info about writing Sources/Sinks, but it is somewhat out of date, and doesn't reflect the things we've learned recently.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                sisk Stephen Sisk
              • Votes:
                2 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: