Camel
  1. Camel
  2. CAMEL-2563

The Trace mechanism is inflexible and inefficient - specifically it doesn't enable custom tracing around a node.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.3.0
    • Component/s: camel-core
    • Labels:
      None
    • Environment:

      All.

      Description

      What it won't let me do:
      I want to be able to correlate the "out" trace with the "in" trace, in one database row without commiting the row until the route has completed.
      This requires a JPA transaction to exist around each of the nodes that are called.
      I'm finding that the transaction has ended by the time of the "out" trace.

      Inefficiencies:
      1. It causes the construction of the new Exchange object and a bunch of String objects that I don't want.
      2. It causes the invocation of a new route, that is unnecessary.

      I think it would be better to:
      1. Pass the class to use as the TraceInterceptor into Tracer.
      2. Break the existing TraceInterceptor into two, a base class that tracks the RouteNodes and a subclass that implements traceExchange.
      3. Change traceExchange so that it returns an Object and pass that Object in to the call to traceExchange for "out" traces.
      4. Give the Tracer a payload object that can be used to pass information to the TraceInterceptor.
      5. Change the various trace* functions to take in and return a payload Object.
      Custom users could then dervice from TraceInterceptor and override the trace* functions as necessary (probably just traceExchange).

      I'm working on a patch.

      1. tracechanges.diff
        6 kB
        Yaytay
      2. tracechanges.diff
        18 kB
        Yaytay
      3. tracechanges.diff
        27 kB
        Yaytay
      4. tracechanges.diff
        67 kB
        Yaytay
      5. tracechanges.diff
        68 kB
        Yaytay

        Activity

        Hide
        Christian Müller added a comment -

        Jim, currently we are also planed to implement our own TraceInterceptor which fits all our needs. We have two requirements which the current implementation doesn't fit:

        • (1) We want to limit the traced services. We do not want to see the calls to splitter(), jaxb(), .... May be we could include and/or exclude services by its name per regExp configuration.
        • (2) We want provide our own JPAEntity which also logs the headers and attachments in its own tables.
        • (3) Because of (2), the TraceInterceptor needs more time to persist the data and should be executed in its own thread. -> The TraceInterceptor should not slow down the performance of our services.

        Could you also have an eye on that?
        ... and if you need help, please let me know...

        Regards,
        Christian

        Show
        Christian Müller added a comment - Jim, currently we are also planed to implement our own TraceInterceptor which fits all our needs. We have two requirements which the current implementation doesn't fit: (1) We want to limit the traced services. We do not want to see the calls to splitter(), jaxb(), .... May be we could include and/or exclude services by its name per regExp configuration. (2) We want provide our own JPAEntity which also logs the headers and attachments in its own tables. (3) Because of (2), the TraceInterceptor needs more time to persist the data and should be executed in its own thread. -> The TraceInterceptor should not slow down the performance of our services. Could you also have an eye on that? ... and if you need help, please let me know... Regards, Christian
        Hide
        Yaytay added a comment -

        Attached is a diff of the changes to Trace and TraceInterceptor.
        I haven't done unit tests for it yet, just tested it using the use I have for it - I'll sort this in a bit.
        I just wanted to include this now to show the scope of what I was trying to do, which is to give the freedom to use your own subclass of the TraceInterceptor.

        Show
        Yaytay added a comment - Attached is a diff of the changes to Trace and TraceInterceptor. I haven't done unit tests for it yet, just tested it using the use I have for it - I'll sort this in a bit. I just wanted to include this now to show the scope of what I was trying to do, which is to give the freedom to use your own subclass of the TraceInterceptor.
        Hide
        Yaytay added a comment -

        Christian,

        My patch should make it simpler for you to create your own TraceInterceptor, but doesn't do anything to provide a TraceInterceptor that meets your needs.
        I think that's the right approach for the core, your requirements should not be baked into the general TraceInterceptor.
        There's possibly an argument for putting the filtering into the general TraceInterceptor, but I think even that is best put into a specific TraceInterceptor.

        Jim

        Show
        Yaytay added a comment - Christian, My patch should make it simpler for you to create your own TraceInterceptor, but doesn't do anything to provide a TraceInterceptor that meets your needs. I think that's the right approach for the core, your requirements should not be baked into the general TraceInterceptor. There's possibly an argument for putting the filtering into the general TraceInterceptor, but I think even that is best put into a specific TraceInterceptor. Jim
        Hide
        Yaytay added a comment -

        Oh, there is also already a filter Predicate that you can set on the Tracer object.
        It's written as though it expects to filter based on the exchange, rather than the current node, but I don't think there is anything to stop you using exchange.getUnitOfWork().getTracedRouteNodes() to get the details of the node and filtering on that.

        Show
        Yaytay added a comment - Oh, there is also already a filter Predicate that you can set on the Tracer object. It's written as though it expects to filter based on the exchange, rather than the current node, but I don't think there is anything to stop you using exchange.getUnitOfWork().getTracedRouteNodes() to get the details of the node and filtering on that.
        Hide
        Claus Ibsen added a comment -

        Just look at this page
        http://camel.apache.org/advanced-configuration-of-camelcontext-using-spring.html

        Then you can do as this example and be in full control how you want to do your custom tracing et all.

        Also the tracer has a destination which you can send to a SEDA queue to have it executed in another thread etc.

        And if you want to do some custom logic in your own Tracer then subclass the existing Tracer and do you custom logic.
        Its easy to use/configure as documented
        http://camel.apache.org/tracer

        Show
        Claus Ibsen added a comment - Just look at this page http://camel.apache.org/advanced-configuration-of-camelcontext-using-spring.html Then you can do as this example and be in full control how you want to do your custom tracing et all. Also the tracer has a destination which you can send to a SEDA queue to have it executed in another thread etc. And if you want to do some custom logic in your own Tracer then subclass the existing Tracer and do you custom logic. Its easy to use/configure as documented http://camel.apache.org/tracer
        Hide
        Claus Ibsen added a comment -

        @Jim

        Good you got started on the patch.

        I think the approach should be to extend the Tracer and thus what I would look for is to for example

        • maybe some methods in existing Tracer should promote from private to protected
        • maybe there are some places a strategy callback method should be added so extended classes can do custom logic

        Then you just configure to use your Tracer as shown on the link above, eg just add a spring bean and Camel will use your class as the Tracer instead of the default one.

        Show
        Claus Ibsen added a comment - @Jim Good you got started on the patch. I think the approach should be to extend the Tracer and thus what I would look for is to for example maybe some methods in existing Tracer should promote from private to protected maybe there are some places a strategy callback method should be added so extended classes can do custom logic Then you just configure to use your Tracer as shown on the link above, eg just add a spring bean and Camel will use your class as the Tracer instead of the default one.
        Hide
        Claus Ibsen added a comment -

        @Christian

        Good idea about the async behavior of the Tracer, even though you could do that by sending to a seda endpoint, but then that means adding an additional route, which you may not like.

        But again by having the existing Tracer a bit more easy to extend with strategy callbacks allows you to do your own async persist to JPA as you like.
        We could also add a option so Camel could let the existing send to destination occur async.

        And the idea with a filter to skip tracing certain nodes is also a great idea. For starters we could add a boolean shouldTrace(ProcessorDefinition node, Exchange exchange) method which you can implement your own logic to indicate whether or not it should be traced.

        Then later we can add a regexp kind pattern to the default Tracer to have that out of the box.

        Its all a fair bit of work and hence we love contributions. But I have to stress that we must have unit tests for this as the Tracer is actually more complex to get working correctly in all the odd size combinations of routes there is out there. I know we got a few JIRA tickets with issues currently.

        Show
        Claus Ibsen added a comment - @Christian Good idea about the async behavior of the Tracer, even though you could do that by sending to a seda endpoint, but then that means adding an additional route, which you may not like . But again by having the existing Tracer a bit more easy to extend with strategy callbacks allows you to do your own async persist to JPA as you like. We could also add a option so Camel could let the existing send to destination occur async. And the idea with a filter to skip tracing certain nodes is also a great idea. For starters we could add a boolean shouldTrace(ProcessorDefinition node, Exchange exchange) method which you can implement your own logic to indicate whether or not it should be traced. Then later we can add a regexp kind pattern to the default Tracer to have that out of the box. Its all a fair bit of work and hence we love contributions. But I have to stress that we must have unit tests for this as the Tracer is actually more complex to get working correctly in all the odd size combinations of routes there is out there. I know we got a few JIRA tickets with issues currently.
        Hide
        Yaytay added a comment -

        @Claus

        One of the things I'm keen to retain is the route node tracking of the built-in TraceInterceptor (I tried working without that and ended up replicating it!).
        So I think the solution is to try to keep the built in Trace, but allow it to work with a custom TraceInterceptor (that derives from TraceInterceptor and thus keeps the route node tracking).
        If you want integration with Spring maybe the solution is to register a TraceInterceptorFactory and give that to Trace.

        I've got to go away for the weekend, so more next week.

        Thanks.

        Jim

        Show
        Yaytay added a comment - @Claus One of the things I'm keen to retain is the route node tracking of the built-in TraceInterceptor (I tried working without that and ended up replicating it!). So I think the solution is to try to keep the built in Trace, but allow it to work with a custom TraceInterceptor (that derives from TraceInterceptor and thus keeps the route node tracking). If you want integration with Spring maybe the solution is to register a TraceInterceptorFactory and give that to Trace. I've got to go away for the weekend, so more next week. Thanks. Jim
        Hide
        Christian Müller added a comment -

        @Claus, Jim,
        thanks for your feedback, the useful links and letting me know, that our requirements are not unusual.
        At present I don't know, how I could support your work. May be if we have a more concrete solution, I could write the unit tests? Let me know, if you have other ideas in your mind.
        Thanks,
        Christian

        Show
        Christian Müller added a comment - @Claus, Jim, thanks for your feedback, the useful links and letting me know, that our requirements are not unusual. At present I don't know, how I could support your work. May be if we have a more concrete solution, I could write the unit tests? Let me know, if you have other ideas in your mind. Thanks, Christian
        Hide
        Yaytay added a comment -

        I'm attaching a rather better patch.

        Changes:

        • There is now a TraceInterceptorFactory interface with a default implementation DefaultTraceInterceptorFactory.
        • The Tracer class has get/setTraceInterceptorFactory.
        • Tracer has got separate functions for performing the actual exchange, with In and Out exchange traces identified.
        • There is a unit test (three ).

        Todo:

        • Not looked at configuring it from Spring yet.
        • Not sure whether the segregate into separate functions for the exchange should be behind an interface rather than just a subclass.

        Thoughts on Christian's requirements:
        1. Filtering by node is complicated by the fact that the node you receive as an interceptor may be another interceptor.
        My unit test walks the tree of DelegateProcessors to find the real node, but that will only work if the interceptors are DelegateProcessors, and there is nothing to ensure that they are.
        2. Using custom JPA code works, that's precisely what I want to do.
        3. I'm not sure of the logic of doing tracing in a separate thread - it's a sacrifice of maximum throughput for reduced latency.
        Whatever you pass to the separate thread will have to be a self contained copy, which with attachments could be large (unless you can make the attachments imutable objects?).
        It's made easier with my changes, but I haven't done anything towards it as such.

        Show
        Yaytay added a comment - I'm attaching a rather better patch. Changes: There is now a TraceInterceptorFactory interface with a default implementation DefaultTraceInterceptorFactory. The Tracer class has get/setTraceInterceptorFactory. Tracer has got separate functions for performing the actual exchange, with In and Out exchange traces identified. There is a unit test (three ). Todo: Not looked at configuring it from Spring yet. Not sure whether the segregate into separate functions for the exchange should be behind an interface rather than just a subclass. Thoughts on Christian's requirements: 1. Filtering by node is complicated by the fact that the node you receive as an interceptor may be another interceptor. My unit test walks the tree of DelegateProcessors to find the real node, but that will only work if the interceptors are DelegateProcessors, and there is nothing to ensure that they are. 2. Using custom JPA code works, that's precisely what I want to do. 3. I'm not sure of the logic of doing tracing in a separate thread - it's a sacrifice of maximum throughput for reduced latency. Whatever you pass to the separate thread will have to be a self contained copy, which with attachments could be large (unless you can make the attachments imutable objects?). It's made easier with my changes, but I haven't done anything towards it as such.
        Hide
        Yaytay added a comment -

        @Claus,

        I've added a setDefaultTracer to CamelContext, so you can create your own Tracer in Spring, set properties on it and then set is as the default.

        However I think what you are suggesting is the creation of a separate object to do the tracing, so a tracing singleton can be set up in Spring and all the trace calls go via it.
        I'm happy to do that if that's what you mean.

        What are your thoughts on the complications of filtering by node?

        Show
        Yaytay added a comment - @Claus, I've added a setDefaultTracer to CamelContext, so you can create your own Tracer in Spring, set properties on it and then set is as the default. However I think what you are suggesting is the creation of a separate object to do the tracing, so a tracing singleton can be set up in Spring and all the trace calls go via it. I'm happy to do that if that's what you mean. What are your thoughts on the complications of filtering by node?
        Hide
        Yaytay added a comment -

        Another version.

        Extracts the traceExchange methods into a separate interface (TraceHandler) and provides a default implementation (DefaultTraceHandler).
        A single instance of the TraceHandler can be set on the Tracer and will be used for all routes created from then on.
        The DefaultTraceHandler takes the JpaTraceEventMessageClass as a String argument, which allows custom JPA classes to be used easily.
        The disadvantage of using the TraceHandler (as opposed to deriving from TraceInterceptor and replacing that) is that you are working with a single object across all nodes, routes and threads - if you want to have a instance data per node you should derive from TraceInterceptor and use the TraceInterceptorFactory.

        The DefaultTraceHandler throws up an interesting issue:
        It has to load the JpaTraceEventMessageClass dynamically, because it doesn't know at compile time whether or not it should be used.
        If that is done in a TraceHandler then it needs to synchronize on every call (which could be an issue).
        However now users could manually request a JpaTraceHandler, which could statically load the JpaTraceEventMessageClass - it might even be possible to be generic based on the class - but I couldn't make this the default because it would be a breaking change.
        Maybe the best approach is to make the default TraceInterceptor a subclass of the DefaultTraceInterceptor that implements the existing behaviour and does not use the TraceHandler at all.
        Then if users want to use the TraceHandler mechanism they manually specify the parent class TraceInterceptor (or the factory for it) which does use the TraceHandler.
        It sounds complex, but it's just a few lines in Spring.

        Need more tests, and still considering the issue of filtering by node - suggestions appreciated.
        Any thoughts welcomed.

        Show
        Yaytay added a comment - Another version. Extracts the traceExchange methods into a separate interface (TraceHandler) and provides a default implementation (DefaultTraceHandler). A single instance of the TraceHandler can be set on the Tracer and will be used for all routes created from then on. The DefaultTraceHandler takes the JpaTraceEventMessageClass as a String argument, which allows custom JPA classes to be used easily. The disadvantage of using the TraceHandler (as opposed to deriving from TraceInterceptor and replacing that) is that you are working with a single object across all nodes, routes and threads - if you want to have a instance data per node you should derive from TraceInterceptor and use the TraceInterceptorFactory. The DefaultTraceHandler throws up an interesting issue: It has to load the JpaTraceEventMessageClass dynamically, because it doesn't know at compile time whether or not it should be used. If that is done in a TraceHandler then it needs to synchronize on every call (which could be an issue). However now users could manually request a JpaTraceHandler, which could statically load the JpaTraceEventMessageClass - it might even be possible to be generic based on the class - but I couldn't make this the default because it would be a breaking change. Maybe the best approach is to make the default TraceInterceptor a subclass of the DefaultTraceInterceptor that implements the existing behaviour and does not use the TraceHandler at all. Then if users want to use the TraceHandler mechanism they manually specify the parent class TraceInterceptor (or the factory for it) which does use the TraceHandler. It sounds complex, but it's just a few lines in Spring. Need more tests, and still considering the issue of filtering by node - suggestions appreciated. Any thoughts welcomed.
        Hide
        Claus Ibsen added a comment -

        Nice you keep momentum on this improvements Jim.

        I would suggest to focus on making it easier to plugin your custom tracer.
        And then create a new ticket for the filtering of nodes. Then we can look at that later.

        Then its also easier to have accepted as the patch is smaller, more focused on one problem, etc.

        Show
        Claus Ibsen added a comment - Nice you keep momentum on this improvements Jim. I would suggest to focus on making it easier to plugin your custom tracer. And then create a new ticket for the filtering of nodes. Then we can look at that later. Then its also easier to have accepted as the patch is smaller, more focused on one problem, etc.
        Hide
        Yaytay added a comment -

        @Claus,
        Thanks, fair point.

        I've pulled back slightly from the previous patch to make it slightly tighter, but I need more unit tests for it.
        When it's done the options will be (in order of increasing complexity):
        1. Use the standard v2.2 features, these will be completely unchanged in behaviour and won't be implemented using my new features.
        This guarantees backward compatibility and makes the change a bit smaller.
        2. Specify your own JpaTraceEventMessageClassName as a property on the Tracer.
        This gives easy freedom of the database structure/naming and allows you to add new fields that can be calculated without reference to the exchange (the host name is one example).
        This does not give you the ability to query the Exchange object and still uses the v2.2 infrastructure.
        3. Write a handler class that implements the TraceHandler interface and specify it as a property on the Tracer.
        This gives direct access to the Exchange object with no other infrastructure, the downside is that the TraceHandler is shared across all routes/nodes/threads.
        4. Write a handler class and a TraceInterceptorFactory object, then in the TraceInterceptorFactory create a new TraceInterceptor (using the out of the box class) but after creation add your own TraceHandler instance.
        This sounds complex, but is really simple and allows you to create your own TraceHandler instance per node, providing direct access to the Exchange with no threading worries and without risk of breaking the TraceInterceptor.
        5. Write your own subclass of the TraceInterceptor and a TraceInterceptorFactory object that instantiates it.
        This provide the ultimate control, but plenty of opportunity for breaking the TraceInterceptor.

        All of these should be fully configurable from Spring - my holdup now is that I need tests for all of them and I'm not clear on how to test spring configurations from the camel unit tests.

        Show
        Yaytay added a comment - @Claus, Thanks, fair point. I've pulled back slightly from the previous patch to make it slightly tighter, but I need more unit tests for it. When it's done the options will be (in order of increasing complexity): 1. Use the standard v2.2 features, these will be completely unchanged in behaviour and won't be implemented using my new features. This guarantees backward compatibility and makes the change a bit smaller. 2. Specify your own JpaTraceEventMessageClassName as a property on the Tracer. This gives easy freedom of the database structure/naming and allows you to add new fields that can be calculated without reference to the exchange (the host name is one example). This does not give you the ability to query the Exchange object and still uses the v2.2 infrastructure. 3. Write a handler class that implements the TraceHandler interface and specify it as a property on the Tracer. This gives direct access to the Exchange object with no other infrastructure, the downside is that the TraceHandler is shared across all routes/nodes/threads. 4. Write a handler class and a TraceInterceptorFactory object, then in the TraceInterceptorFactory create a new TraceInterceptor (using the out of the box class) but after creation add your own TraceHandler instance. This sounds complex, but is really simple and allows you to create your own TraceHandler instance per node, providing direct access to the Exchange with no threading worries and without risk of breaking the TraceInterceptor. 5. Write your own subclass of the TraceInterceptor and a TraceInterceptorFactory object that instantiates it. This provide the ultimate control, but plenty of opportunity for breaking the TraceInterceptor. All of these should be fully configurable from Spring - my holdup now is that I need tests for all of them and I'm not clear on how to test spring configurations from the camel unit tests.
        Hide
        Yaytay added a comment -

        Here it is, my final diff (subject to comments).

        The options for customisation are as stated in my previous comment.
        The changes to the camel-core main classes are pretty minor, the vast bulk of the patch is tests.
        There are tests of three different types of tracing for each of the customisation options when configured with java, and then for just one of the types of tracing when configured by spring (for each customisation option).

        Jim

        Show
        Yaytay added a comment - Here it is, my final diff (subject to comments). The options for customisation are as stated in my previous comment. The changes to the camel-core main classes are pretty minor, the vast bulk of the patch is tests. There are tests of three different types of tracing for each of the customisation options when configured with java, and then for just one of the types of tracing when configured by spring (for each customisation option). Jim
        Hide
        Claus Ibsen added a comment -

        You have to grant rights to Apache on the files attached, otherwise we cannot use them.

        Show
        Claus Ibsen added a comment - You have to grant rights to Apache on the files attached, otherwise we cannot use them.
        Hide
        Yaytay added a comment -

        Here it is, my final diff reloaded with the correct license (subject to comments).

        The options for customisation are as stated in my previous comment.
        The changes to the camel-core main classes are pretty minor, the vast bulk of the patch is tests.
        There are tests of three different types of tracing for each of the customisation options when configured with java, and then for just one of the types of tracing when configured by spring (for each customisation option).

        Jim

        Show
        Yaytay added a comment - Here it is, my final diff reloaded with the correct license (subject to comments). The options for customisation are as stated in my previous comment. The changes to the camel-core main classes are pretty minor, the vast bulk of the patch is tests. There are tests of three different types of tracing for each of the customisation options when configured with java, and then for just one of the types of tracing when configured by spring (for each customisation option). Jim
        Hide
        Claus Ibsen added a comment -

        Jim

        If possible could you run checkstyle to ensure your patch has no CS errors.

        See more here about Checkstyle
        http://camel.apache.org/building.html

        Show
        Claus Ibsen added a comment - Jim If possible could you run checkstyle to ensure your patch has no CS errors. See more here about Checkstyle http://camel.apache.org/building.html
        Hide
        Yaytay added a comment -

        'tis done.
        I think you know there were lots of failures that I've fixed in this diff.
        Sorry I didn't pick up on that earlier.

        Show
        Yaytay added a comment - 'tis done. I think you know there were lots of failures that I've fixed in this diff. Sorry I didn't pick up on that earlier.
        Hide
        Claus Ibsen added a comment -

        trunk: 932351.

        Finally got a bit time to get this patch into the trunk.
        I do polish the patch a bit and renamed the TraceHandler to TraceEventHandler.

        Jim we need to have the documentation update with more details about this. Do you want to help out?
        http://camel.apache.org/tracer

        Show
        Claus Ibsen added a comment - trunk: 932351. Finally got a bit time to get this patch into the trunk. I do polish the patch a bit and renamed the TraceHandler to TraceEventHandler. Jim we need to have the documentation update with more details about this. Do you want to help out? http://camel.apache.org/tracer
        Hide
        Yaytay added a comment -

        Will do.
        It might take me a couple of days but I'll submit a draft.

        Show
        Yaytay added a comment - Will do. It might take me a couple of days but I'll submit a draft.
        Hide
        Yaytay added a comment -

        Claus,
        Just noticed that you removed the exceptionFromProcess handling to capture an exception during a process and make that available to the trace and I'm wondering why.
        I put that in there because the tracer is occuring before the exception is handled (and added to the exchange), so I don't think there is any way to know that a process has failed without it - have I got that wrong?

        Show
        Yaytay added a comment - Claus, Just noticed that you removed the exceptionFromProcess handling to capture an exception during a process and make that available to the trace and I'm wondering why. I put that in there because the tracer is occuring before the exception is handled (and added to the exchange), so I don't think there is any way to know that a process has failed without it - have I got that wrong?
        Hide
        Claus Ibsen added a comment -

        The javadoc explains where the exception is.

        eg as in standard Camel: exchange.getException()

        Show
        Claus Ibsen added a comment - The javadoc explains where the exception is. eg as in standard Camel: exchange.getException()
        Hide
        Yaytay added a comment -

        Well, what on Earth was going on in my head when I added the exceptionFromProcess then?
        For some reason I wasn't (or didn't think I was) getting anything from either getException or Exchange.getProperty()
        No matter, I can see it's working from the test.
        Thanks for removing that waste of bits.

        Show
        Yaytay added a comment - Well, what on Earth was going on in my head when I added the exceptionFromProcess then? For some reason I wasn't (or didn't think I was) getting anything from either getException or Exchange.getProperty() No matter, I can see it's working from the test. Thanks for removing that waste of bits.
        Hide
        Claus Ibsen added a comment -

        I have updated the wiki page to include the new options
        https://cwiki.apache.org/confluence/display/CAMEL/Tracer

        Show
        Claus Ibsen added a comment - I have updated the wiki page to include the new options https://cwiki.apache.org/confluence/display/CAMEL/Tracer
        Hide
        Claus Ibsen added a comment -

        Closing all resolved tickets from 2010 or older

        Show
        Claus Ibsen added a comment - Closing all resolved tickets from 2010 or older

          People

          • Assignee:
            Claus Ibsen
            Reporter:
            Yaytay
          • Votes:
            2 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 20h
              20h
              Remaining:
              Remaining Estimate - 20h
              20h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development