Details
-
Wish
-
Status: Resolved
-
Major
-
Resolution: Won't Fix
-
None
-
None
-
None
-
Guru
Description
Wish
The wish is for a transaction log store in Camel to provide detailed tracking and performance metrics on end-to-end transactions across multiple exchanges/routes, storing metadata, exchange data, and performance metrics in a configurable, persistent registry to better monitor business processes.
Background
A lot of data in tracing and observability is on the exchange level. How many exchanges are completed or failed, for example. In many cases, what I really want to know if a transaction is successful or failed.
With a transaction I mean that the complete lifecycle of a message on Camel, this
may contain many routes and exchanges. For example, when Seda or ActiveMQ is used
a new exchange is created, but it's still the same transaction.
To have metrics and more insights on transactions, I would like to have a transaction log (through a separate registry?). This is a separate store with information about the transaction.
Implementation ideas
Below are few ideas to implement this wish:
1. Initialize the store
Setup a transaction log/store (either in memory or external store) through a registry. This is to enable it and to configure the store.
2. Transaction item
A transaction item is a definition of a specific transaction. It may contain the following items:
- Name: The name to indicate the transaction. Like "order" or "invoice". This is so to say the overarching name of the integration/business process.
- StartingPoint: A routeId or groupId where the transaction starts (triggers a new transaction).
- CorrelationID(s): One or more correlationId (headers) that can be set just like a header/exchangePropery/variable, this contains for example the orderID that is set on the first step of the transaction.
3. Tracking transactions
A transaction is probably best tracked by something like a breadcrumbId. Preferably a transactionId. At least something that cannot be edited by users (I sometimes see people change the breadcrumbId, to separate transactions into multiple segments).
By default, only the start and end of a transaction is tracked. To get more fine-grained control, one could also track intermediate points like a route, a step, or a node.
The transaction is tracked during processing, and then finished on completion. The whole counts as one "UnitOfWork".
4. The transaction store
The transaction store contains all the tracked items. Maybe there could be divided into separate segments like:
- metadata
- name (name of the transaction)
- description (description of type of transaction)
- transactionId
- transactionHistory (similar to messageHistory)
- correlationId(s)
- timestamps
- exceptions
- exchange data
- headers, properties, variables
- bodyType
- body
etc
- performance metrics
- load (CPU), memory (RAM), threads
- number of transactions (completed, pending, failed)
- throughput
- response time
- total process time of an transaction
- average
- etc
It should be possible to configure which of these items the user what want to store. By default, only metadata.
5. Transaction API's
There need to be an API to configure and operate the transaction store. (including JMX), similar to backlogtracer.
Production
Some of the solutions I saw are only available during development or test, but the need is especially big for production environment. A solution needs to keep in mind that this would run for billion of messages. The flexibility to turn it easily on and off.
For example, by default, the body of a message doesn't need to be stored. Or only the first 32kb need to be stored. But for specific case users, one to enable this on the fly for a specific transaction.
Transactions are probably needed persistent, so either need a messagestore (disk) similar to brokers or an external database. I'm not sure what the best stores are for this kind of data. A normal SQL database like Postgres, a search database like Elastic, a metrics database like Prometheus or key/value store like Redis?
End users
The idea is that end users have a good insight in what is going on in the integration layer. So that they can question like the following:
- How many order transactions are processed?
- Give me a specific order by orderId that custom says he didn't receive?
- How many failed messages are there? Which ones did go wrong, where did it go wrong, and what is the exact error?
- Can we replay the failed order?
- Where is the bottleneck on the system
Conclusion
This is just an idea/wish, but something I see coming back from the start of my career as integration specialist (around 2007), until today.
Most consultancy firms or end customers have built something themselves, I see with names like "Track & Trace", "MessageStore", "ESB Collector", "Transactions", "Control center".
I also built a solution myself, and are happy to share this solution, or others I saw. A more generic solution to monitor (track & trace) transactions of business processes would be very welcome.
