[BEAM-14431] Handle nulls using SnowflakeIO

• Type: Bug
• Status: Open
• Priority: P2
• Resolution: Unresolved
• Affects Version/s: 2.38.0
• Component/s: io-java-snowflake
• Labels: snowflake

Language

Python

• Assignee: Unassigned
• Reporter: Varun
• Created: 06/May/22 07:48
• Updated: 05/Jun/22 01:10

Description

When SnowflakeIO is used within python beam code using ReadFromSnowflake [1] ; the returned data isn't able to list the nulls from snowflake side.

The nulls from snowflake becomes b'N' ; which is a wrong value and gives incorrect value for the column. In case of strings the actual value can also be 'N'. So it becomes tough to handle it in csv_mapper.

To mitigate this we are using query instead of table and using ifnull / coleasce within the select query for String columns. For non-string columns we are converting back b'N' to null or empty as per requirement.

 

[1] : https://beam.apache.org/releases/pydoc/2.33.0/apache_beam.io.snowflake.html#apache_beam.io.snowflake.ReadFromSnowflake