Camel
  1. Camel
  2. CAMEL-2853

camel-stream - tailing logfile does not seem to work when logfile is rolled over

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Later
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.4.0
    • Component/s: camel-stream
    • Labels:
      None

      Description

      Should support something like how tail --retry works, basically to try and re-open a file if it gets completely overwritten or deleted/rewritten.

        Activity

        Hide
        Stan Lewis added a comment -

        Sending src/main/java/org/apache/camel/component/stream/StreamConsumer.java
        Sending src/test/java/org/apache/camel/component/stream/ScanStreamFileTest.java
        Transmitting file data ..
        Committed revision 957691.

        Show
        Stan Lewis added a comment - Sending src/main/java/org/apache/camel/component/stream/StreamConsumer.java Sending src/test/java/org/apache/camel/component/stream/ScanStreamFileTest.java Transmitting file data .. Committed revision 957691.
        Hide
        Stan Lewis added a comment -

        Should mention you get this behavior now when you specify scanStream=true.

        Show
        Stan Lewis added a comment - Should mention you get this behavior now when you specify scanStream=true.
        Hide
        Rob Howes added a comment -

        Although the benefit in being able to retry overwritten files is extremely useful, this change also has resulted in the side effect of having scanStream now continually loop over the existing file.

        I would have expected that the improvement result in retrying the file only on being overwritten and not on reaching the EOS, and based on the description below would say that the change made is not implemented as intended.

        Worth noting that I have only tried this on Windows and have not confirmed the behaviour on UNIX.

        Show
        Rob Howes added a comment - Although the benefit in being able to retry overwritten files is extremely useful, this change also has resulted in the side effect of having scanStream now continually loop over the existing file. I would have expected that the improvement result in retrying the file only on being overwritten and not on reaching the EOS, and based on the description below would say that the change made is not implemented as intended. Worth noting that I have only tried this on Windows and have not confirmed the behaviour on UNIX.
        Hide
        Stan Lewis added a comment -

        Will have a look, the test doesn't appear to show this behavior if I adjust it so it expects an exact number of items...

        Show
        Stan Lewis added a comment - Will have a look, the test doesn't appear to show this behavior if I adjust it so it expects an exact number of items...
        Hide
        Rob Howes added a comment -

        I have created a simple test project that demonstrates the issue and attached.

        Again, I have only run this in Windows but see a clear difference when switching from:

        • Camel 2.4.0 - continuous iteration over the same 2 lines
        • Camel 2.3.0 - only iterates once over the 2 lines
        Show
        Rob Howes added a comment - I have created a simple test project that demonstrates the issue and attached. Again, I have only run this in Windows but see a clear difference when switching from: Camel 2.4.0 - continuous iteration over the same 2 lines Camel 2.3.0 - only iterates once over the 2 lines
        Hide
        Stan Lewis added a comment -

        Unfortunately it looks like this feature may have to wait for Java 7 to be fully realized, as there's no reliable way to detect when a file has been deleted/recreated without having to resort to JNI. Using a WatchService would make it possible to implement this, see here - http://java.sun.com/developer/technicalArticles/javase/nio/#6

        Show
        Stan Lewis added a comment - Unfortunately it looks like this feature may have to wait for Java 7 to be fully realized, as there's no reliable way to detect when a file has been deleted/recreated without having to resort to JNI. Using a WatchService would make it possible to implement this, see here - http://java.sun.com/developer/technicalArticles/javase/nio/#6
        Hide
        Rob Howes added a comment -

        Ah ok - thanks for looking into this.

        Perhaps it would be worth adding an attribute retry on StreamEndpoint that defaults to false that preserves the pre-Camel 2.4.0 behaviour of scanStream?

        This way people upgrading don't have to make any changes and those users who need the retry logic can set the flag to true.

        Show
        Rob Howes added a comment - Ah ok - thanks for looking into this. Perhaps it would be worth adding an attribute retry on StreamEndpoint that defaults to false that preserves the pre-Camel 2.4.0 behaviour of scanStream? This way people upgrading don't have to make any changes and those users who need the retry logic can set the flag to true.
        Hide
        Stan Lewis added a comment -

        Yeah, I like that, will work on getting that added

        Show
        Stan Lewis added a comment - Yeah, I like that, will work on getting that added
        Hide
        Stan Lewis added a comment -

        Committed r1068896
        M components/camel-stream/src/test/java/org/apache/camel/component/stream/ScanStreamFileTest.java
        M components/camel-stream/src/main/java/org/apache/camel/component/stream/StreamConsumer.java
        M components/camel-stream/src/main/java/org/apache/camel/component/stream/StreamEndpoint.java

        Now I'll just update the wiki

        Show
        Stan Lewis added a comment - Committed r1068896 M components/camel-stream/src/test/java/org/apache/camel/component/stream/ScanStreamFileTest.java M components/camel-stream/src/main/java/org/apache/camel/component/stream/StreamConsumer.java M components/camel-stream/src/main/java/org/apache/camel/component/stream/StreamEndpoint.java Now I'll just update the wiki
        Hide
        Rob Howes added a comment -

        Thanks for this patch - works great.

        Show
        Rob Howes added a comment - Thanks for this patch - works great.
        Hide
        Stan Lewis added a comment -

        Cool! I'll probably create another Jira to keep track of the fact that this should be revisited in the future

        Show
        Stan Lewis added a comment - Cool! I'll probably create another Jira to keep track of the fact that this should be revisited in the future

          People

          • Assignee:
            Stan Lewis
            Reporter:
            Stan Lewis
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development