[HOP-4239] Consider a binary file data type

• Type: Improvement
• Status: Closed
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: Migrated to GHI
• Component/s: API, Pipelines
• Assignee: Unassigned
• Reporter: Matt Casters
• Created: 23/Sep/22 10:26
• Updated: 15/Feb/23 08:58
• Resolved: 15/Feb/23 08:58

Description

We could consider a new value type called "Binary file" which stores large data sets in (temporary) files instead of keeping data in memory.

This would solve use-cases where large amounts of data need to be transferred from one location to another.  

For example, Oracle BLOB fields have the ability to contain up to 4GB or data.  That is more than fits in a Binary field and it would be a shame to waste memory on it in the first place.

When writing the data to another database or location you can stream the data back out from the file.

The trick would be that we would need a way to clean the temporary files back up at the end of a pipeline.  The constructor of the ValueMetaBinaryFile() would need to have a hook to a mechanism to remove the file when no longer needed.