Details
Major Description
People have data stored in Hive tables. Currently these tables do not support Spark streaming, so customers do not have a good way to natively stream this data in Spark. The current solutions involve an intermediary to track which data has been read and periodically execute batch jobs. This use case should be supported by Spark's in-built streaming mechanism.
From doing some research, Hive supports streaming https://cwiki.apache.org/confluence/display/Hive/Streaming+Data+Ingest+V2 but Spark does not support streaming on tables in Hive format. I don't think it makes sense to start copying Hive server-side code into Spark, but we could copy the relevant logic and wrap it in the DataSourceV2 APIs to enable this feature. To not break backwards compatibility, we would probably want to gate this behind a new Spark property.