[KAFKA-2365] Copycat checklist

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0.0
• Fix Version/s: 0.10.0.0
• Component/s: connect
• Labels: feature
• Assignee: Ewen Cheslack-Postava
• Reporter: Ewen Cheslack-Postava
• Reviewer: Ewen Cheslack-Postava
• Created: 27/Jul/15 05:40
• Updated: 15/Jun/16 16:53
• Resolved: 15/Jun/16 16:53

Description

This covers the development plan for KIP-26. There are a number of features that can be developed in sequence to make incremental progress, and often in parallel:

• Initial patch - connector API and core implementation
• Runtime data API
• Standalone CLI
• REST API
• Distributed copycat - CLI
• Distributed copycat - coordinator
• Distributed copycat - config storage
• Distributed copycat - offset storage
• Log/file connector (sample source/sink connector)
• Elasticsearch sink connector (sample sink connector for full log -> Kafka -> Elasticsearch sample pipeline)
• Copycat metrics
• System tests (including connector tests)
• Mirrormaker connector
• Copycat documentation

This is an initial list, but it might need refinement to allow for more incremental progress and may be missing features we find we want before the initial release.

Sub-Tasks

• Initial patch for Copycat: Resolved
• Add Copycat runtime data API: Resolved
• Add Copycat standalone CLI: Resolved
• Add Copycat REST API: Resolved
• Add pause/unpause connector support: Resolved
• Add distributed coordinator implementation for Copycat: Resolved
• Copycat distributed config storage: Resolved
• Copycat distributed offset storage: Resolved
• Implement Copycat log/file connector: Resolved
• Implement elasticsearch Copycat sink connector: Resolved
• Add copycat system tests: Resolved
• Add Copycat documentation: Resolved
• Allow copycat connector plugins to be aliased to simpler names: Resolved
• Add caching for converted Copycat schemas in JSONConverter: Resolved
• Reduce copycat configs to only specify a converter or serializer, not both: Resolved
• Define logical types for Copycat data API: Resolved
• Add CopycatExceptions to indicate transient and permanent errors in a connector/task: Resolved
• Handle non-CopycatExceptions from SinkTasks: Resolved
• Allow copycat sinks to request periodic invocation of put even if no new data is available: Resolved
• Allow copycat sink tasks to pause/resume consumption of specific topic partitions: Resolved
• Add schema projection utilities: Resolved
• Add Time logical type for Copycat: Resolved
• Null offsets in copycat causes exception in OffsetStorageWriter: Resolved
• Copycat worker should not call connector's/task's start methods in the control thread: Resolved
• Connectors and tasks should have versions that can be reported when they are instantiated: Resolved
• Rename Copycat -> Kafka Connect: Resolved