Uploaded image for project: 'Camel'
  1. Camel
  2. CAMEL-15955

camel-core - API and SPI for Camel components to resume from offset

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 3.12.0
    • camel-core
    • None
    • Unknown

    Description

      We need a API and SPI for components to optionally support some way of offset management. Meaning that a component is able to resume from a given offset to continue from last known state. For example if a Camel application was stopped/crashed, then it can restart and resume from last known offset.

      Something like Apache Kafka offset has.

      This benefits Camel in general, and Camel Kafka Connector also.

      An use-case is the famous big files. So the file component is reading a file, and process it line by line via a splitter. Then we would like to store an offset how far it has processed. And in case of crash/restart etc then it can continue from last known position. Today it would restart from top.

      Same for SQL,JDBC components with SQL ResultSet and so on.

      The offset management should support different quality levels and a SPI so different implementations can plugin. One could be to store to a plain file and for each step. And other can have a batch to store after N rows or N seconds. Or like in Kafka with every N seconds.

      Then each Camel component that can support this, can then implement the API and logic for handling this. For the file component we can maybe store an offset in the input stream, so when resuming we then forward the stream to that position.

      Attachments

        Issue Links

          relates to

          New Feature - A new feature of the product, which has yet to be developed. CAMEL-16751 camel-core - API and SPI for Transactional Batch Consumer

          • Major - Major loss of function.
          • Open

          New Feature - A new feature of the product, which has yet to be developed. CAMEL-15562 repository & strategy to store and retrieve the latest state of a consumer and allow to resume upon restart

          • Major - Major loss of function.
          • Resolved

          New Feature - A new feature of the product, which has yet to be developed. CAMEL-17026 camel-kafka: add resume support on the producer

          • Major - Major loss of function.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. CAMEL-17045 camel-file: add support for the resume strategy

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              orpiske Otavio Rodolfo Piske
              davsclaus Claus Ibsen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: