Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-6611

A Python Sink for BigQuery with File Loads in Streaming

    XMLWordPrintableJSON

    Details

      Description

      The Java SDK supports a bunch of methods for writing data into BigQuery, while the Python SDK supports the following:

      Qucik and dirty early design doc: https://s.apache.org/beam-bqfl-py-streaming
      The Java SDK also supports File Loads for Streaming pipelines see BatchLoads application.

      File loads have the advantage of being much cheaper than streaming inserts (although they also are slower for the records to show up in the table).

        Attachments

          Activity

            People

            • Assignee:
              ttanay Tanay Tummalapalli
              Reporter:
              pabloem Pablo Estrada
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 10h 50m
                10h 50m