[GSOC-140] Apache ShardingSphere: Add ShardingSphere Kafka source connector

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: ShardingSphere full-time gsoc2023 mentor
• Assignee: Unassigned
• Reporter: Hongsheng Zhong
• Created: 18/Feb/23 14:24
• Updated: 27/Mar/23 08:40

Description

Apache ShardingSphere

Apache ShardingSphere is positioned as a Database Plus, and aims at building a standard layer and ecosystem above heterogeneous databases. It focuses on how to reuse existing databases and their respective upper layer, rather than creating a new database. The goal is to minimize or eliminate the challenges caused by underlying databases fragmentation.

Page: https://shardingsphere.apache.org
Github: https://github.com/apache/shardingsphere 

Background

The community just added CDC (change data capture) feature recently. Change feed will be published in created network connection after logging in, then it could be consumed.

Since Kafka is popular distributed event streaming platform, it's useful to import change feed into Kafka for later processing.

Task

1. Familiar with ShardingSphere CDC client usage, create publication and subscribe change feed.
2. Familiar with Kafka connector development, develop source connector, integrate with ShardingSphere CDC. Persist change feed to Kafka topics properly.
3. Add unit test and E2E integration test.

Relevant Skills

1. Java language
2. Basic knowledge of CDC and Kafka
3. Maven

References

• https://github.com/apache/shardingsphere/issues/22500
• https://kafka.apache.org/documentation/#connect_development
• https://github.com/apache/kafka/tree/trunk/connect/file/src
• https://github.com/confluentinc/kafka-connect-jdbc

Local Test Steps

1. Modify `conf/server.yaml`, uncomment `cdc-server-port: 33071` to enable CDC. (Refer to step 2)
2. Configure proxy, refer to `Prerequisites` and `Procedure` in build to configure proxy (Newer version could be used too, current stable version is 5.3.1).
3. Start proxy server, it'll start CDC server too.
4. Download ShardingSphere source code from https://github.com/apache/shardingsphere , modify and run `org.apache.shardingsphere.data.pipeline.cdc.client.example.Bootstrap`. It'll print `records:` by default in `Bootstrap`.
5. Execute some ISNERT/UPDATE/DELETE SQLs in proxy to generate change feed, and then check in `Bootstrap` console.

Mentor

Hongsheng Zhong, PMC of Apache ShardingSphere, [e-mail]

Xinze Guo, Committer of Apache ShardingSphere, [e-mail]