Major Description
Apache SeaTunnel(Incubating)
SeaTunnel is a very easy-to-use ultra-high-performance distributed data integration platform that supports real-time synchronization of massive data. It can synchronize tens of billions of data stably and efficiently every day, and has been used in the production of nearly 100 companies.
SeaTunnel provides a Connector API that does not depend on a specific execution engine. Connectors (Source, Transform, Sink) developed based on this API can run On many different engines, such as SeaTunnel Zeta, Flink, Spark that are currently supported. SeaTunnel has supported more than 100 Connectors, and the number is surging.
Website: https://seatunnel.apache.org/
GitHub: https://github.com/apache/incubator-seatunnel
Background
To use SeaTunnel, the current user needs to first create and write a config file that specifies the engine that runs the job, as well as engine related parameters. Then define the Source, Transform, and Sink of the job. We hope to provide a client that allows users to define the engine, Source, Transform, and Sink information of the job run directly through code in the client without having to start with a config file. The user can then submit the job definition information through the Client, and SeaTunnel will run these jobs. After the job is submitted, the user can obtain the status of the job running through the Client. For jobs that are already running, users can use this client to manage them, such as stopping jobs, temporary jobs, and so on.
Task
1. Discuss with the mentors what you need to do
2. Learn the details of the Apache SeaTunnel project
3. Discuss and complete design and development
Relevant Skills
- Familiar with Java, Http
- Familiar with SeaTunnel is better
Mentor
- Mentor: Jun Gao, Apache SeaTunnel(Incubating) PPMC Member, gaojun2048@apache.org
- Mentor: Li Liu, Apache SeaTunnel(Incubating) Commiter, ic4y@apache.org
- Mailing List: dev@seatunnel.apache.org