[FLUME-2938] JDBC Source

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.8.0
• Component/s: Sinks+Sources
• Assignee: Unassigned
• Reporter: Lior Zeno
• Created: 24/Jun/16 14:08
• Updated: 01/Jun/22 15:27

Description

The idea is to allow migrating data from SQL stores to NoSQL stores or HDFS for archiving purposes.
This source will get a statement to execute and a scheduling policy. It will be able to fetch timestamped data by performing range queries on a configurable field (this can fetch data with incremental id as well). For fault-tolerance, the last fetched value can be checkpointed to a file.
Dealing with large datasets can be done via the fetch_size parameter. (Ref: https://docs.oracle.com/cd/A87860_01/doc/java.817/a83724/resltse5.htm)