Uploaded image for project: 'Kafka'
  1. Kafka
  2. KAFKA-2365

Copycat checklist

    XMLWordPrintableJSON

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0.0
    • Fix Version/s: 0.10.0.0
    • Component/s: KafkaConnect
    • Labels:

      Description

      This covers the development plan for KIP-26. There are a number of features that can be developed in sequence to make incremental progress, and often in parallel:

      • Initial patch - connector API and core implementation
      • Runtime data API
      • Standalone CLI
      • REST API
      • Distributed copycat - CLI
      • Distributed copycat - coordinator
      • Distributed copycat - config storage
      • Distributed copycat - offset storage
      • Log/file connector (sample source/sink connector)
      • Elasticsearch sink connector (sample sink connector for full log -> Kafka -> Elasticsearch sample pipeline)
      • Copycat metrics
      • System tests (including connector tests)
      • Mirrormaker connector
      • Copycat documentation

      This is an initial list, but it might need refinement to allow for more incremental progress and may be missing features we find we want before the initial release.

        Attachments

        1.
        Initial patch for Copycat Sub-task Resolved Ewen Cheslack-Postava  
        2.
        Add Copycat runtime data API Sub-task Resolved Ewen Cheslack-Postava  
        3.
        Add Copycat standalone CLI Sub-task Resolved Ewen Cheslack-Postava  
        4.
        Add Copycat REST API Sub-task Resolved Ewen Cheslack-Postava  
        5.
        Add pause/unpause connector support Sub-task Resolved Jason Gustafson  
        6.
        Add distributed coordinator implementation for Copycat Sub-task Resolved Ewen Cheslack-Postava  
        7.
        Copycat distributed config storage Sub-task Resolved Ewen Cheslack-Postava  
        8.
        Copycat distributed offset storage Sub-task Resolved Ewen Cheslack-Postava  
        9.
        Implement Copycat log/file connector Sub-task Resolved Ewen Cheslack-Postava  
        10.
        Implement elasticsearch Copycat sink connector Sub-task Resolved Liquan Pei  
        11.
        Add copycat system tests Sub-task Resolved Ewen Cheslack-Postava  
        12.
        Add Copycat documentation Sub-task Resolved Ewen Cheslack-Postava  
        13.
        Allow copycat connector plugins to be aliased to simpler names Sub-task Resolved Gwen Shapira  
        14.
        Add caching for converted Copycat schemas in JSONConverter Sub-task Resolved Ewen Cheslack-Postava  
        15.
        Reduce copycat configs to only specify a converter or serializer, not both Sub-task Resolved Ewen Cheslack-Postava  
        16.
        Define logical types for Copycat data API Sub-task Resolved Ewen Cheslack-Postava  
        17.
        Add CopycatExceptions to indicate transient and permanent errors in a connector/task Sub-task Resolved Liquan Pei  
        18.
        Handle non-CopycatExceptions from SinkTasks Sub-task Resolved Ewen Cheslack-Postava  
        19.
        Allow copycat sinks to request periodic invocation of put even if no new data is available Sub-task Resolved Liquan Pei  
        20.
        Allow copycat sink tasks to pause/resume consumption of specific topic partitions Sub-task Resolved Ewen Cheslack-Postava  
        21.
        Add schema projection utilities Sub-task Resolved Liquan Pei  
        22.
        Add Time logical type for Copycat Sub-task Resolved Ewen Cheslack-Postava

        0%

        Original Estimate - 1h
        Remaining Estimate - 1h
        23.
        Null offsets in copycat causes exception in OffsetStorageWriter Sub-task Resolved Ewen Cheslack-Postava  
        24.
        Copycat worker should not call connector's/task's start methods in the control thread Sub-task Resolved Ewen Cheslack-Postava  
        25.
        Connectors and tasks should have versions that can be reported when they are instantiated Sub-task Resolved Ewen Cheslack-Postava  
        26.
        Rename Copycat -> Kafka Connect Sub-task Resolved Ewen Cheslack-Postava  

          Activity

            People

            • Assignee:
              ewencp Ewen Cheslack-Postava
              Reporter:
              ewencp Ewen Cheslack-Postava
              Reviewer:
              Ewen Cheslack-Postava
            • Votes:
              1 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 1h
                1h
                Remaining:
                Remaining Estimate - 1h
                1h
                Logged:
                Time Spent - Not Specified
                Not Specified