[BEAM-9896] Add streaming for SnowflakeIO.Write to Java SDK

• Type: New Feature
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.24.0
• Fix Version/s: 2.24.0
• Component/s: io-ideas
• Assignee: Kasia Kucharczyk
• Reporter: Dariusz Aniszewski
• Created: 06/May/20 14:57
• Updated: 13/Apr/23 11:02
• Resolved: 15/May/21 17:59

Description

SnowflakeIO has currently only batch processing. To provide streaming processing it is required to use SnowPipe which enables streaming write to Snowflake. Currently it is impossible to provide streaming read.

Snowflake ingest library for Java provides ingest methods which provides using SnowPipe and loading data by authorising with key-pair and providing staged files (ex. on GCS). It also provides method retrieving load history which is the only way to check if files could be loaded properly. Checking history can be only provided by actively requesting API because loading data is asynchronous.

Issue Links

• is a child of: BEAM-9893 Add SnowflakeIO to Java SDK
• links to: GitHub Pull Request #12151