Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-9896

Add streaming for SnowflakeIO.Write to Java SDK

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: P2
    • Resolution: Unresolved
    • Affects Version/s: 2.24.0
    • Fix Version/s: 2.24.0
    • Component/s: io-ideas
    • Labels:
      None

      Description

      SnowflakeIO has currently only batch processing. To provide streaming processing it is required to use SnowPipe which enables streaming write to Snowflake. Currently it is impossible to provide streaming read.

      Snowflake ingest library for Java provides ingest methods which provides using SnowPipe and loading data by authorising with key-pair and providing staged files (ex. on GCS). It also provides method retrieving load history which is the only way to check if files could be loaded properly. Checking history can be only provided by actively requesting API because loading data is asynchronous.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kasiak Kasia Kucharczyk
                Reporter:
                dariusz.aniszewski Dariusz Aniszewski
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 4h 50m
                  4h 50m