Camel
  1. Camel
  2. CAMEL-5155

Support JCR Component as Consumer

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.9.1
    • Fix Version/s: 2.10.0
    • Component/s: camel-jcr
    • Labels:
      None
    • Estimated Complexity:
      Unknown

      Description

      I tried to use jcr component as consumer, expecting jcr observation event listener like behavior, but I realized that it's not supported yet. The current jcr component supports producer only. So, I'm considering to implement JcrConsumer an contribute a patch.

      My idea is as follows:
      i) the uri format is the same. e.g. 'jcr://user:password@repository/absolute/path/from/node'; the node path is used for javax.jcr.observation.ObservationManager#addEventListener(..., absPath, ...).
      ii) additionally, we need parameters for eventTypes, isDeep, uuid[], nodeTypeName[], noLocal parameters (See #addEventListener() for detail).
      For example, we may probably use somethings like 'jcr://user:password@repository/absolute/path/from/node?nodeTypeName=demo:news&eventTypes=3&isDeep=true.
      iii) JcrConsumer registers JCR observation event listener from the specified uri information.
      iv) When events notified, I think it can simply return a list of javax.jcr.observation.Event objects in IN message.

      1. jcr-consumer-patch.txt
        34 kB
        Woonsan Ko
      2. jcr-consumer-patch-2.txt
        34 kB
        Woonsan Ko

        Activity

        Hide
        Woonsan Ko added a comment -

        Cool!
        Also, the use cases you showed with the example (as well as r1332008) are very interesting and useful!

        Cheers,

        Woonsan

        Show
        Woonsan Ko added a comment - Cool! Also, the use cases you showed with the example (as well as r1332008) are very interesting and useful! Cheers, Woonsan
        Hide
        Bilgin Ibryam added a comment -

        I updated the documentation and even blogged an example if its usage http://bit.ly/M34OiG

        Show
        Bilgin Ibryam added a comment - I updated the documentation and even blogged an example if its usage http://bit.ly/M34OiG
        Hide
        Woonsan Ko added a comment -

        Hi Bilgin,

        My actual use case is actually very simple. I've developed a data import tool [1] for a JCR repository product. Currently, I configure file2 component url to start the import tool at runtime in order to import xml files to jcr nodes. So, the file2 consumer is just a signal for the import tool execution. Now, I want to use jcr node(s) as signals instead of files, which helps me consolidate the management user interface into CMS UI and avoid thinking of clustering issues. My local workspace already contains jcr consumer configurations; I will commit that upon a new camel release with the feature.

        Regarding the use case of producer reading events, I agree that users probably want to do more such as retrieving nodes from the events and doing some manipulation actions.
        But, I guess they need to weave some custom components with the consumer in most cases. For example, one of typical scenarios is that they might want to create summary data nodes when a news article document node is added/modified/deleted. Then a custom component, in the pipeline with the consumer, creates a jcr session from the repository, reads event infos, queries data from the repository and creates a summary data node.
        Maybe we'll find a good use case for the idea later. e.g., simple logging node creation on news document events. Anyway, more generic options support with the producer might be needed. e.g., how to change primary node type for a new created node, how to change node creation paths dynamically, etc.

        Regards, Woonsan

        [1] http://import-tool.forge.onehippo.org/

        Show
        Woonsan Ko added a comment - Hi Bilgin, My actual use case is actually very simple. I've developed a data import tool [1] for a JCR repository product. Currently, I configure file2 component url to start the import tool at runtime in order to import xml files to jcr nodes. So, the file2 consumer is just a signal for the import tool execution. Now, I want to use jcr node(s) as signals instead of files, which helps me consolidate the management user interface into CMS UI and avoid thinking of clustering issues. My local workspace already contains jcr consumer configurations; I will commit that upon a new camel release with the feature. Regarding the use case of producer reading events, I agree that users probably want to do more such as retrieving nodes from the events and doing some manipulation actions. But, I guess they need to weave some custom components with the consumer in most cases. For example, one of typical scenarios is that they might want to create summary data nodes when a news article document node is added/modified/deleted. Then a custom component, in the pipeline with the consumer, creates a jcr session from the repository, reads event infos, queries data from the repository and creates a summary data node. Maybe we'll find a good use case for the idea later. e.g., simple logging node creation on news document events. Anyway, more generic options support with the producer might be needed. e.g., how to change primary node type for a new created node, how to change node creation paths dynamically, etc. Regards, Woonsan [1] http://import-tool.forge.onehippo.org/
        Hide
        Bilgin Ibryam added a comment -

        You documented pretty well the java classes, I will add them to the wiki. The latest jcr code is in trunk, have a look here http://camel.apache.org/how-can-i-get-the-source-code.html

        I was thinking how this jcr consumer would be helpful, and once you got a list of jcr Event, and for example if it is an update/add operation, the next thing user wants to do would be to retrieve this node from jcr in order to do some action with it. So I think extending the jcr producer with ability to read nodes by a given id/path from the event might be helpful. WDYT?

        Also are you using currently the Consumer, I wonder what is your use case to listen only for events, w/o retrieving the actual nodes?

        Show
        Bilgin Ibryam added a comment - You documented pretty well the java classes, I will add them to the wiki. The latest jcr code is in trunk, have a look here http://camel.apache.org/how-can-i-get-the-source-code.html I was thinking how this jcr consumer would be helpful, and once you got a list of jcr Event, and for example if it is an update/add operation, the next thing user wants to do would be to retrieve this node from jcr in order to do some action with it. So I think extending the jcr producer with ability to read nodes by a given id/path from the event might be helpful. WDYT? Also are you using currently the Consumer, I wonder what is your use case to listen only for events, w/o retrieving the actual nodes?
        Hide
        Woonsan Ko added a comment -

        Hi Bilgin,

        Thank you very much!
        By the way, where can I find the source of the JCR component documentation? It doesn't look having xdoc or something similar in the trunk.
        Please let me know if I may help the documentation as well.

        Kind regards,

        Woonsan

        Show
        Woonsan Ko added a comment - Hi Bilgin, Thank you very much! By the way, where can I find the source of the JCR component documentation? It doesn't look having xdoc or something similar in the trunk. Please let me know if I may help the documentation as well. Kind regards, Woonsan
        Hide
        Bilgin Ibryam added a comment -

        Woonsan,

        I committed your patch, but will keep the ticket open till the documentation is updated.

        Thanks again,

        Show
        Bilgin Ibryam added a comment - Woonsan, I committed your patch, but will keep the ticket open till the documentation is updated. Thanks again,
        Hide
        Woonsan Ko added a comment -

        Hi Richard / Bilgin,

        I've just uploaded a new improved patch (jcr-consumer-patch-2.txt) for the following, based on your suggestions:

        • removing unnecessary call on #unregisterListenerAndLogoutSession() on non-live jcr session (see Bilgin's comment).
        • replacing the background thread by ScheduledExecutorService via ExecutorServiceManager#newSingleThreadScheduledExecutor().
        • removing try/catch in JcrMessage.toString().

        Thank a lot for your reviews!

        Cheers,

        Woonsan

        Show
        Woonsan Ko added a comment - Hi Richard / Bilgin, I've just uploaded a new improved patch (jcr-consumer-patch-2.txt) for the following, based on your suggestions: removing unnecessary call on #unregisterListenerAndLogoutSession() on non-live jcr session (see Bilgin's comment). replacing the background thread by ScheduledExecutorService via ExecutorServiceManager#newSingleThreadScheduledExecutor(). removing try/catch in JcrMessage.toString(). Thank a lot for your reviews! Cheers, Woonsan
        Hide
        Woonsan Ko added a comment -

        Posting a new improved patch (jcr-consumer-patch-2.txt), adopting Bilgin's and Richard's suggestions.

        Show
        Woonsan Ko added a comment - Posting a new improved patch (jcr-consumer-patch-2.txt), adopting Bilgin's and Richard's suggestions.
        Hide
        Woonsan Ko added a comment -

        Hi Bilgin,

        > why are you calling unregisterListenerAndLogoutSession when there is no live session?
        >
        > if (!isSessionLive && isObservationListeningAllowed())

        { unregisterListenerAndLogoutSession(); ... }

        Thanks for the sharp review!
        I think the call on #unregisterListenerAndLogoutSession() in the if block can be removed.
        Yes, you're right. It was unnecessary to call it for non live session.
        So, the if block ("if (!isSessionLive && isObservationListeningAllowed())

        { ... }

        ") can contain only the try block of #createSessionAndRegisterListener() call.

        Kind regards,

        Woonsan

        Show
        Woonsan Ko added a comment - Hi Bilgin, > why are you calling unregisterListenerAndLogoutSession when there is no live session? > > if (!isSessionLive && isObservationListeningAllowed()) { unregisterListenerAndLogoutSession(); ... } Thanks for the sharp review! I think the call on #unregisterListenerAndLogoutSession() in the if block can be removed. Yes, you're right. It was unnecessary to call it for non live session. So, the if block ("if (!isSessionLive && isObservationListeningAllowed()) { ... } ") can contain only the try block of #createSessionAndRegisterListener() call. Kind regards, Woonsan
        Hide
        Richard Kettelerij added a comment -

        Hi Woonsan,

        I've had a look at your patch and noticed you're starting a new thread in the JcrConsumer class. When starting new threads in Camel it's advisable to use the ExecutorServiceManager. This allows users to plugin different threading implementation (e.g. JCA WorkManager or CommonJ).

        One minor detail: the try/catch in JcrMessage.toString() isn't necessary.

        Kind regards,
        Richard

        Show
        Richard Kettelerij added a comment - Hi Woonsan, I've had a look at your patch and noticed you're starting a new thread in the JcrConsumer class. When starting new threads in Camel it's advisable to use the ExecutorServiceManager . This allows users to plugin different threading implementation (e.g. JCA WorkManager or CommonJ). One minor detail: the try/catch in JcrMessage.toString() isn't necessary. Kind regards, Richard
        Hide
        Bilgin Ibryam added a comment -

        Hi Woonsan,

        I'm looking at your patch, great job.
        I have question though: in JcrConsumer.JcrConsumerSessionListenerChecker.run()

        why are you calling unregisterListenerAndLogoutSession when there is no live session?

        if (!isSessionLive && isObservationListeningAllowed())

        { unregisterListenerAndLogoutSession(); ... }

        Bilgin

        Show
        Bilgin Ibryam added a comment - Hi Woonsan, I'm looking at your patch, great job. I have question though: in JcrConsumer.JcrConsumerSessionListenerChecker.run() why are you calling unregisterListenerAndLogoutSession when there is no live session? if (!isSessionLive && isObservationListeningAllowed()) { unregisterListenerAndLogoutSession(); ... } Bilgin
        Hide
        Woonsan Ko added a comment -

        Here's the detail of my implementation with the patch:

        • Assumption/Constraint: A (remote) JCR server may restart at any time. So the JCR session, which is used to (un)register observation listener, should be checked if it is still live periodically and re-initialized unless it is live. This is the main difference from the producer implementation which creates a new session on demand whenever it needs to create nodes from the message.
        • JcrEndpoint
        • The following parameters (properties) were added: eventTypes, deep, uuids, nodeTypeNames, noLocal.
          See the javadoc of those new properties for detail. Those are needed to register jcr observation listeners.
          Also, sessionLiveCheckIntervalOnStart and sessionLiveCheckInterval parameters were added for periodic session live checking.
        • #createConsumer() now creates and returns an instance of new JcrConsumer class.
        • JcrConsumer
        • Overrides the lifecycle methods.
        • Session and listener are created asynchronously in a separate session live checker thread.
          The session live checker thread is responsible for creating session and registering listener. Also, it is responsible for checking if the session is still live and re-initializing the session and re-registering listener if necessary.
          This approach is more adequate for the cases where JCR repository server is separately deployed or managed.
        • JcrConsumer registers a JCR observation listener named 'EndpointEventListener'.
        • EndpointEventListener
        • This JCR observation listener implementation receives JCR events, creates an exchange and process the exchange.
        • JcrMessage
        • Camel message implementation for JCR events.
        • The message body is created as a List from the JCR events iterator for easy expression based integration.
        • JcrConsumerTest
        • Unit test for JcrConsumer
        • Creates a route, "jcr://user:pass@repository/home/test?eventTypes=1, which will be re-routed to "direct:a".
        • A test thread ("JcrConsumerThread") receives from "direct:a".
        • Adds a node to /home/test/node to trigger the route ("jcr://...").
        • The test thread should receive the message.
        • Wait the thread stopping and reads the exchange to verify.
        • The old JcrRouteTest.java was divided to JcrProducerTest.java and JcrRouteTestSupport.java just for simple refactoring (e.g., JcrConsumerTest shared the some logic such as repository initialization).
        • For information, I tested with the following spring route configuration in my test project:

        <route>
        <from uri="jcr://user:pass@repository/import-application/inbox/signal?eventTypes=3&deep=true&synchronous=false" />
        <to uri="direct:execute-import-application" />
        </route>

        Where can I find the source of the JCR component documentation? I'd like to contribute the documentation, too, if applied.

        Kind regards,

        Woonsan

        Show
        Woonsan Ko added a comment - Here's the detail of my implementation with the patch: Assumption/Constraint: A (remote) JCR server may restart at any time. So the JCR session, which is used to (un)register observation listener, should be checked if it is still live periodically and re-initialized unless it is live. This is the main difference from the producer implementation which creates a new session on demand whenever it needs to create nodes from the message. JcrEndpoint The following parameters (properties) were added: eventTypes, deep, uuids, nodeTypeNames, noLocal. See the javadoc of those new properties for detail. Those are needed to register jcr observation listeners. Also, sessionLiveCheckIntervalOnStart and sessionLiveCheckInterval parameters were added for periodic session live checking. #createConsumer() now creates and returns an instance of new JcrConsumer class. JcrConsumer Overrides the lifecycle methods. Session and listener are created asynchronously in a separate session live checker thread. The session live checker thread is responsible for creating session and registering listener. Also, it is responsible for checking if the session is still live and re-initializing the session and re-registering listener if necessary. This approach is more adequate for the cases where JCR repository server is separately deployed or managed. JcrConsumer registers a JCR observation listener named 'EndpointEventListener'. EndpointEventListener This JCR observation listener implementation receives JCR events, creates an exchange and process the exchange. JcrMessage Camel message implementation for JCR events. The message body is created as a List from the JCR events iterator for easy expression based integration. JcrConsumerTest Unit test for JcrConsumer Creates a route, "jcr://user:pass@repository/home/test?eventTypes=1, which will be re-routed to "direct:a". A test thread ("JcrConsumerThread") receives from "direct:a". Adds a node to /home/test/node to trigger the route ("jcr://..."). The test thread should receive the message. Wait the thread stopping and reads the exchange to verify. The old JcrRouteTest.java was divided to JcrProducerTest.java and JcrRouteTestSupport.java just for simple refactoring (e.g., JcrConsumerTest shared the some logic such as repository initialization). For information, I tested with the following spring route configuration in my test project: <route> <from uri="jcr://user:pass@repository/import-application/inbox/signal?eventTypes=3&deep=true&synchronous=false" /> <to uri="direct:execute-import-application" /> </route> Where can I find the source of the JCR component documentation? I'd like to contribute the documentation, too, if applied. Kind regards, Woonsan
        Hide
        Woonsan Ko added a comment -

        Attaching a patch implementing JCR Consumer with a unit test.

        Show
        Woonsan Ko added a comment - Attaching a patch implementing JCR Consumer with a unit test.
        Hide
        Woonsan Ko added a comment -

        Claus Ibsen:
        """
        This sounds like a really good idea.

        We love contributions, and would look very much forward to your help
        with this component.
        http://camel.apache.org/contributing.html

        I like the fact if the uri can be the same / almost the same.

        Also when registering listeners in JCR, then remember to cleanup when
        the JcrConsumer stops, in the doStop method.
        There should be some API in JCR to unregister the listener as well.
        """

        Woonsan:
        "Thank you very much for the valuable input!
        Yes, JCR has an API to unregister listener, so I will need to do proper clean ups.
        I'll keep you updated."

        Show
        Woonsan Ko added a comment - Claus Ibsen: """ This sounds like a really good idea. We love contributions, and would look very much forward to your help with this component. http://camel.apache.org/contributing.html I like the fact if the uri can be the same / almost the same. Also when registering listeners in JCR, then remember to cleanup when the JcrConsumer stops, in the doStop method. There should be some API in JCR to unregister the listener as well. """ Woonsan: "Thank you very much for the valuable input! Yes, JCR has an API to unregister listener, so I will need to do proper clean ups. I'll keep you updated."

          People

          • Assignee:
            Bilgin Ibryam
            Reporter:
            Woonsan Ko
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development