Details
-
New Feature
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
Description
For Azure blob storage, similar to HUDI-1897
GH issue request
Since Streaming ingestion using DeltaStreamer from DFS that contains parquet files has a problem as explained in this blog here https://medium.com/apache-hudi-blogs/reliable-ingestion-from-aws-s3-using-hudi-b7d5590c78a9 here, I'm working on implementing a similar setup described in the above mentioned blog for ingesting parquet files stored in Azure blob storage and enable event triggers to Azure storage queue.
Currently, hudi-utilities/sources contains support only for S3 events source (S3EventsSource.java) and incremental pulls from S3 (S3EventsHoodieIncrSource.java). The ingestion pattern doesn't seem to support equivalent Azure cloud stack.
Attachments
Issue Links
- relates to
-
HUDI-1897 Implement DeltaStreamer Source for AWS S3
- Closed