Qpid
  1. Qpid
  2. QPID-3121

Cluster management inconsistency when using persistent store.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9
    • Fix Version/s: 0.9
    • Component/s: C++ Clustering
    • Labels:
      None

      Description

      If cluster_tests.py, test_management is modified to enable durable messages, it fails the log comparison test shows messages like this on one broker but not the others:

      trace Changed V1 statistics org.apache.qpid.broker:connection:127.0.0.1:52742-127.0.0.1:44104 len=NN
      trace Changed V2 statistics org.apache.qpid.broker:connection:127.0.0.1:52742-127.0.0.1:44104

      To date this hasn't been seen to actually cause a cluster crash but in principle it is possible it could.

      To reproduce, build the message store at: http://anonsvn.jboss.org/repos/rhmessaging/store/
      In the tests/cluster directory, run this in a loop:

      make check TESTS=run_python_cluster_tests CLUSTER_TESTS='.test_management -DDURATION=2'

      It will fail, usually on the first iteration, showing the log files that don't match. Use diff or other such tool to confirm that the mismatched lines are as above. The file may also contain some other mismatches showing a different number of stats in a periodic update - that is a consequence of the above.

        Activity

        Hide
        Alan Conway added a comment -

        Modified version of cluster_tests.py to enable durable messages for test_management.

        Show
        Alan Conway added a comment - Modified version of cluster_tests.py to enable durable messages for test_management.
        Hide
        Alan Conway added a comment -

        Fix posted to review board https://reviews.apache.org/r/473/

        Show
        Alan Conway added a comment - Fix posted to review board https://reviews.apache.org/r/473/
        Hide
        Justin Ross added a comment -

        Approved for 0.10. Also reviewed on reviewboard by Ken Giusti.

        Show
        Justin Ross added a comment - Approved for 0.10. Also reviewed on reviewboard by Ken Giusti.
        Hide
        Alan Conway added a comment -

        Fix committed to trunk r1078947

        Show
        Alan Conway added a comment - Fix committed to trunk r1078947
        Hide
        Alan Conway added a comment -

        Also comitted to 0.10 release branch QPID-3121

        Show
        Alan Conway added a comment - Also comitted to 0.10 release branch QPID-3121
        Hide
        Alan Conway added a comment -

        Committed to trunk and 0.10 branch.

        Show
        Alan Conway added a comment - Committed to trunk and 0.10 branch.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/943/
        -----------------------------------------------------------

        Review request for Gordon Sim and Kenneth Giusti.

        Summary
        -------

        QPID-3121: Bug 682815 - Cluster management inconsistency when using persistent store.

        With the recent changes to asynchronous completion, a message can be
        completed either in a journal thread or in the connection thread. If
        it is completed in the connection thread then completeRcvMsg is called
        immediately in the connection thread. Otherwise completeRcvMsg is
        called via requestIOProcessing as an IO callback.

        This makes the ordering of management events generated during
        completeRcvMsg unpredictalbe and causes an inconsistency error when
        completeRcvMsg updates connection stats.

        The fix is to mark completeRcvMsg as a cluster-unsafe scope so no management
        messages will be generated regardless of how it is called.

        This addresses bug QPID-3121.
        https://issues.apache.org/jira/browse/QPID-3121

        Diffs


        /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1138296

        Diff: https://reviews.apache.org/r/943/diff

        Testing
        -------

        Thanks,

        Alan

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/943/ ----------------------------------------------------------- Review request for Gordon Sim and Kenneth Giusti. Summary ------- QPID-3121 : Bug 682815 - Cluster management inconsistency when using persistent store. With the recent changes to asynchronous completion, a message can be completed either in a journal thread or in the connection thread. If it is completed in the connection thread then completeRcvMsg is called immediately in the connection thread. Otherwise completeRcvMsg is called via requestIOProcessing as an IO callback. This makes the ordering of management events generated during completeRcvMsg unpredictalbe and causes an inconsistency error when completeRcvMsg updates connection stats. The fix is to mark completeRcvMsg as a cluster-unsafe scope so no management messages will be generated regardless of how it is called. This addresses bug QPID-3121 . https://issues.apache.org/jira/browse/QPID-3121 Diffs /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1138296 Diff: https://reviews.apache.org/r/943/diff Testing ------- Thanks, Alan
        Hide
        Alan Conway added a comment -

        Proposed fix up for review at https://reviews.apache.org/r/943/

        Show
        Alan Conway added a comment - Proposed fix up for review at https://reviews.apache.org/r/943/
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/943/#review885
        -----------------------------------------------------------

        /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp
        <https://reviews.apache.org/r/943/#comment1926>

        The fix looks fine, but the comment is wrong. completeRcvMsg() isn't called by the journal thread directly. completeRcvMsg() should only execute on an IO thread. The journal thread will call IncompleteIngressMsgXfer::completed(), which schedules a call to completeRcvMsg() on an available IO thread.

        If you're seeing completeRcvMsg() being called from the Journal thread, that's a bug!

        • Kenneth

        On 2011-06-22 18:39:13, Alan Conway wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/943/

        -----------------------------------------------------------

        (Updated 2011-06-22 18:39:13)

        Review request for Gordon Sim and Kenneth Giusti.

        Summary

        -------

        QPID-3121: Bug 682815 - Cluster management inconsistency when using persistent store.

        With the recent changes to asynchronous completion, a message can be

        completed either in a journal thread or in the connection thread. If

        it is completed in the connection thread then completeRcvMsg is called

        immediately in the connection thread. Otherwise completeRcvMsg is

        called via requestIOProcessing as an IO callback.

        This makes the ordering of management events generated during

        completeRcvMsg unpredictalbe and causes an inconsistency error when

        completeRcvMsg updates connection stats.

        The fix is to mark completeRcvMsg as a cluster-unsafe scope so no management

        messages will be generated regardless of how it is called.

        This addresses bug QPID-3121.

        https://issues.apache.org/jira/browse/QPID-3121

        Diffs

        -----

        /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1138296

        Diff: https://reviews.apache.org/r/943/diff

        Testing

        -------

        Thanks,

        Alan

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/943/#review885 ----------------------------------------------------------- /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp < https://reviews.apache.org/r/943/#comment1926 > The fix looks fine, but the comment is wrong. completeRcvMsg() isn't called by the journal thread directly. completeRcvMsg() should only execute on an IO thread. The journal thread will call IncompleteIngressMsgXfer::completed(), which schedules a call to completeRcvMsg() on an available IO thread. If you're seeing completeRcvMsg() being called from the Journal thread, that's a bug! Kenneth On 2011-06-22 18:39:13, Alan Conway wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/943/ ----------------------------------------------------------- (Updated 2011-06-22 18:39:13) Review request for Gordon Sim and Kenneth Giusti. Summary ------- QPID-3121 : Bug 682815 - Cluster management inconsistency when using persistent store. With the recent changes to asynchronous completion, a message can be completed either in a journal thread or in the connection thread. If it is completed in the connection thread then completeRcvMsg is called immediately in the connection thread. Otherwise completeRcvMsg is called via requestIOProcessing as an IO callback. This makes the ordering of management events generated during completeRcvMsg unpredictalbe and causes an inconsistency error when completeRcvMsg updates connection stats. The fix is to mark completeRcvMsg as a cluster-unsafe scope so no management messages will be generated regardless of how it is called. This addresses bug QPID-3121 . https://issues.apache.org/jira/browse/QPID-3121 Diffs ----- /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1138296 Diff: https://reviews.apache.org/r/943/diff Testing ------- Thanks, Alan
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-06-22 18:55:34, Kenneth Giusti wrote:

        > /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp, line 326

        > <https://reviews.apache.org/r/943/diff/1/?file=21538#file21538line326>

        >

        > The fix looks fine, but the comment is wrong. completeRcvMsg() isn't called by the journal thread directly. completeRcvMsg() should only execute on an IO thread. The journal thread will call IncompleteIngressMsgXfer::completed(), which schedules a call to completeRcvMsg() on an available IO thread.

        >

        > If you're seeing completeRcvMsg() being called from the Journal thread, that's a bug!

        No you're right. I updated the comment.

        • Alan

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/943/#review885
        -----------------------------------------------------------

        On 2011-06-22 19:08:26, Alan Conway wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/943/

        -----------------------------------------------------------

        (Updated 2011-06-22 19:08:26)

        Review request for Gordon Sim and Kenneth Giusti.

        Summary

        -------

        QPID-3121: Bug 682815 - Cluster management inconsistency when using persistent store.

        With the recent changes to asynchronous completion, completeRcvMsg can

        be called immediately by the thread that enqueues the message or

        deferred as an IO Callback till after journal threads are done with

        the message.

        The choice between these two options is unpredicatable, so the

        ordering of management events generated during completeRcvMsg

        unpredictalbe and can cause an inconsistency error when completeRcvMsg

        updates connection stats.

        The fix is to mark completeRcvMsg as a cluster-unsafe scope so no

        management messages will be generated regardless of how it is called.

        This addresses bug QPID-3121.

        https://issues.apache.org/jira/browse/QPID-3121

        Diffs

        -----

        /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1138296

        Diff: https://reviews.apache.org/r/943/diff

        Testing

        -------

        Thanks,

        Alan

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-06-22 18:55:34, Kenneth Giusti wrote: > /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp, line 326 > < https://reviews.apache.org/r/943/diff/1/?file=21538#file21538line326 > > > The fix looks fine, but the comment is wrong. completeRcvMsg() isn't called by the journal thread directly. completeRcvMsg() should only execute on an IO thread. The journal thread will call IncompleteIngressMsgXfer::completed(), which schedules a call to completeRcvMsg() on an available IO thread. > > If you're seeing completeRcvMsg() being called from the Journal thread, that's a bug! No you're right. I updated the comment. Alan ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/943/#review885 ----------------------------------------------------------- On 2011-06-22 19:08:26, Alan Conway wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/943/ ----------------------------------------------------------- (Updated 2011-06-22 19:08:26) Review request for Gordon Sim and Kenneth Giusti. Summary ------- QPID-3121 : Bug 682815 - Cluster management inconsistency when using persistent store. With the recent changes to asynchronous completion, completeRcvMsg can be called immediately by the thread that enqueues the message or deferred as an IO Callback till after journal threads are done with the message. The choice between these two options is unpredicatable, so the ordering of management events generated during completeRcvMsg unpredictalbe and can cause an inconsistency error when completeRcvMsg updates connection stats. The fix is to mark completeRcvMsg as a cluster-unsafe scope so no management messages will be generated regardless of how it is called. This addresses bug QPID-3121 . https://issues.apache.org/jira/browse/QPID-3121 Diffs ----- /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1138296 Diff: https://reviews.apache.org/r/943/diff Testing ------- Thanks, Alan
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/943/
        -----------------------------------------------------------

        (Updated 2011-06-22 19:08:26.449120)

        Review request for Gordon Sim and Kenneth Giusti.

        Changes
        -------

        Fixed comment to be more accurate as per Ken's suggestion

        Summary (updated)
        -------

        QPID-3121: Bug 682815 - Cluster management inconsistency when using persistent store.

        With the recent changes to asynchronous completion, completeRcvMsg can
        be called immediately by the thread that enqueues the message or
        deferred as an IO Callback till after journal threads are done with
        the message.

        The choice between these two options is unpredicatable, so the
        ordering of management events generated during completeRcvMsg
        unpredictalbe and can cause an inconsistency error when completeRcvMsg
        updates connection stats.

        The fix is to mark completeRcvMsg as a cluster-unsafe scope so no
        management messages will be generated regardless of how it is called.

        This addresses bug QPID-3121.
        https://issues.apache.org/jira/browse/QPID-3121

        Diffs


        /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1138296

        Diff: https://reviews.apache.org/r/943/diff

        Testing
        -------

        Thanks,

        Alan

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/943/ ----------------------------------------------------------- (Updated 2011-06-22 19:08:26.449120) Review request for Gordon Sim and Kenneth Giusti. Changes ------- Fixed comment to be more accurate as per Ken's suggestion Summary (updated) ------- QPID-3121 : Bug 682815 - Cluster management inconsistency when using persistent store. With the recent changes to asynchronous completion, completeRcvMsg can be called immediately by the thread that enqueues the message or deferred as an IO Callback till after journal threads are done with the message. The choice between these two options is unpredicatable, so the ordering of management events generated during completeRcvMsg unpredictalbe and can cause an inconsistency error when completeRcvMsg updates connection stats. The fix is to mark completeRcvMsg as a cluster-unsafe scope so no management messages will be generated regardless of how it is called. This addresses bug QPID-3121 . https://issues.apache.org/jira/browse/QPID-3121 Diffs /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1138296 Diff: https://reviews.apache.org/r/943/diff Testing ------- Thanks, Alan
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/943/#review887
        -----------------------------------------------------------

        Ship it!

        • Kenneth

        On 2011-06-22 19:08:26, Alan Conway wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/943/

        -----------------------------------------------------------

        (Updated 2011-06-22 19:08:26)

        Review request for Gordon Sim and Kenneth Giusti.

        Summary

        -------

        QPID-3121: Bug 682815 - Cluster management inconsistency when using persistent store.

        With the recent changes to asynchronous completion, completeRcvMsg can

        be called immediately by the thread that enqueues the message or

        deferred as an IO Callback till after journal threads are done with

        the message.

        The choice between these two options is unpredicatable, so the

        ordering of management events generated during completeRcvMsg

        unpredictalbe and can cause an inconsistency error when completeRcvMsg

        updates connection stats.

        The fix is to mark completeRcvMsg as a cluster-unsafe scope so no

        management messages will be generated regardless of how it is called.

        This addresses bug QPID-3121.

        https://issues.apache.org/jira/browse/QPID-3121

        Diffs

        -----

        /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1138296

        Diff: https://reviews.apache.org/r/943/diff

        Testing

        -------

        Thanks,

        Alan

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/943/#review887 ----------------------------------------------------------- Ship it! Kenneth On 2011-06-22 19:08:26, Alan Conway wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/943/ ----------------------------------------------------------- (Updated 2011-06-22 19:08:26) Review request for Gordon Sim and Kenneth Giusti. Summary ------- QPID-3121 : Bug 682815 - Cluster management inconsistency when using persistent store. With the recent changes to asynchronous completion, completeRcvMsg can be called immediately by the thread that enqueues the message or deferred as an IO Callback till after journal threads are done with the message. The choice between these two options is unpredicatable, so the ordering of management events generated during completeRcvMsg unpredictalbe and can cause an inconsistency error when completeRcvMsg updates connection stats. The fix is to mark completeRcvMsg as a cluster-unsafe scope so no management messages will be generated regardless of how it is called. This addresses bug QPID-3121 . https://issues.apache.org/jira/browse/QPID-3121 Diffs ----- /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1138296 Diff: https://reviews.apache.org/r/943/diff Testing ------- Thanks, Alan
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/943/#review896
        -----------------------------------------------------------

        Ship it!

        So to confirm, this method causes the connection stats for outgoing frames and bytes to be updated, and since that can trigger a management message to be sent out, those updates are now disabled? The outgoing frame and byte counts are unlikely to be relied on in any accuracy critical way, so that should be fine.

        • Gordon

        On 2011-06-22 19:08:26, Alan Conway wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/943/

        -----------------------------------------------------------

        (Updated 2011-06-22 19:08:26)

        Review request for Gordon Sim and Kenneth Giusti.

        Summary

        -------

        QPID-3121: Bug 682815 - Cluster management inconsistency when using persistent store.

        With the recent changes to asynchronous completion, completeRcvMsg can

        be called immediately by the thread that enqueues the message or

        deferred as an IO Callback till after journal threads are done with

        the message.

        The choice between these two options is unpredicatable, so the

        ordering of management events generated during completeRcvMsg

        unpredictalbe and can cause an inconsistency error when completeRcvMsg

        updates connection stats.

        The fix is to mark completeRcvMsg as a cluster-unsafe scope so no

        management messages will be generated regardless of how it is called.

        This addresses bug QPID-3121.

        https://issues.apache.org/jira/browse/QPID-3121

        Diffs

        -----

        /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1138296

        Diff: https://reviews.apache.org/r/943/diff

        Testing

        -------

        Thanks,

        Alan

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/943/#review896 ----------------------------------------------------------- Ship it! So to confirm, this method causes the connection stats for outgoing frames and bytes to be updated, and since that can trigger a management message to be sent out, those updates are now disabled? The outgoing frame and byte counts are unlikely to be relied on in any accuracy critical way, so that should be fine. Gordon On 2011-06-22 19:08:26, Alan Conway wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/943/ ----------------------------------------------------------- (Updated 2011-06-22 19:08:26) Review request for Gordon Sim and Kenneth Giusti. Summary ------- QPID-3121 : Bug 682815 - Cluster management inconsistency when using persistent store. With the recent changes to asynchronous completion, completeRcvMsg can be called immediately by the thread that enqueues the message or deferred as an IO Callback till after journal threads are done with the message. The choice between these two options is unpredicatable, so the ordering of management events generated during completeRcvMsg unpredictalbe and can cause an inconsistency error when completeRcvMsg updates connection stats. The fix is to mark completeRcvMsg as a cluster-unsafe scope so no management messages will be generated regardless of how it is called. This addresses bug QPID-3121 . https://issues.apache.org/jira/browse/QPID-3121 Diffs ----- /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1138296 Diff: https://reviews.apache.org/r/943/diff Testing ------- Thanks, Alan
        Hide
        jiraposter@reviews.apache.org added a comment -

        On 2011-06-23 08:48:10, Gordon Sim wrote:

        > So to confirm, this method causes the connection stats for outgoing frames and bytes to be updated, and since that can trigger a management message to be sent out, those updates are now disabled? The outgoing frame and byte counts are unlikely to be relied on in any accuracy critical way, so that should be fine.

        Exactly

        • Alan

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/943/#review896
        -----------------------------------------------------------

        On 2011-06-22 19:08:26, Alan Conway wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/943/

        -----------------------------------------------------------

        (Updated 2011-06-22 19:08:26)

        Review request for Gordon Sim and Kenneth Giusti.

        Summary

        -------

        QPID-3121: Bug 682815 - Cluster management inconsistency when using persistent store.

        With the recent changes to asynchronous completion, completeRcvMsg can

        be called immediately by the thread that enqueues the message or

        deferred as an IO Callback till after journal threads are done with

        the message.

        The choice between these two options is unpredicatable, so the

        ordering of management events generated during completeRcvMsg

        unpredictalbe and can cause an inconsistency error when completeRcvMsg

        updates connection stats.

        The fix is to mark completeRcvMsg as a cluster-unsafe scope so no

        management messages will be generated regardless of how it is called.

        This addresses bug QPID-3121.

        https://issues.apache.org/jira/browse/QPID-3121

        Diffs

        -----

        /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1138296

        Diff: https://reviews.apache.org/r/943/diff

        Testing

        -------

        Thanks,

        Alan

        Show
        jiraposter@reviews.apache.org added a comment - On 2011-06-23 08:48:10, Gordon Sim wrote: > So to confirm, this method causes the connection stats for outgoing frames and bytes to be updated, and since that can trigger a management message to be sent out, those updates are now disabled? The outgoing frame and byte counts are unlikely to be relied on in any accuracy critical way, so that should be fine. Exactly Alan ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/943/#review896 ----------------------------------------------------------- On 2011-06-22 19:08:26, Alan Conway wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/943/ ----------------------------------------------------------- (Updated 2011-06-22 19:08:26) Review request for Gordon Sim and Kenneth Giusti. Summary ------- QPID-3121 : Bug 682815 - Cluster management inconsistency when using persistent store. With the recent changes to asynchronous completion, completeRcvMsg can be called immediately by the thread that enqueues the message or deferred as an IO Callback till after journal threads are done with the message. The choice between these two options is unpredicatable, so the ordering of management events generated during completeRcvMsg unpredictalbe and can cause an inconsistency error when completeRcvMsg updates connection stats. The fix is to mark completeRcvMsg as a cluster-unsafe scope so no management messages will be generated regardless of how it is called. This addresses bug QPID-3121 . https://issues.apache.org/jira/browse/QPID-3121 Diffs ----- /trunk/qpid/cpp/src/qpid/broker/SessionState.cpp 1138296 Diff: https://reviews.apache.org/r/943/diff Testing ------- Thanks, Alan
        Hide
        Alan Conway added a comment -

        Fixed on trunk r1138874

        Show
        Alan Conway added a comment - Fixed on trunk r1138874

          People

          • Assignee:
            Alan Conway
            Reporter:
            Alan Conway
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development