Qpid
  1. Qpid
  2. QPID-4289

[Java Client 0-8/0-9.x] Failover functionality sometimes does not restore the connection

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.16, 0.18, 0.19
    • Fix Version/s: 0.19
    • Component/s: Java Client
    • Labels:
      None

      Description

      Failover functionality on 0-8/0-9.x java client does not restore the connection in the following scenarios:

      • when on initial establishing of the connectivity the connection object is marked as closed after not being able to connect to the first brokers from the broker list but at the end it manages to connect to the one of the following brokers from the broker list
      • there is a racing condition between waking up the connection state waiter and closing the phisical connection on receiving ConnectionClose command from the broker. As result, waiters are notified, new connection is created and closed afterwards.
      • on killing the broker when no IOException is thrown from socket input/output streams but instead -1 is returned by read operation, the failover starts but IoSender thread is left running. As result, on multiple reconnection attempts the thread resources can be exhausted and that can potentially result in OOME.

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open In Progress In Progress
          2m 35s 1 Alex Rudyy 07/Sep/12 13:30
          Resolved Resolved Reopened Reopened
          53d 2h 29m 1 Robbie Gemmell 03/Nov/12 14:20
          Reopened Reopened In Progress In Progress
          1h 1 Robbie Gemmell 03/Nov/12 15:20
          In Progress In Progress Reviewable Reviewable
          2m 55s 2 Robbie Gemmell 03/Nov/12 15:20
          Reviewable Reviewable Resolved Resolved
          8d 13h 55m 2 Philip Harvey 08/Nov/12 05:58
          Philip Harvey made changes -
          Status Ready To Review [ 10006 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Philip Harvey added a comment -

          I reviewed the changes Robbie made in the light of QPID-4344. All looks good.

          Show
          Philip Harvey added a comment - I reviewed the changes Robbie made in the light of QPID-4344 . All looks good.
          Robbie Gemmell made changes -
          Assignee Robbie Gemmell [ gemmellr ] Philip Harvey [ philharveyonline ]
          Hide
          Robbie Gemmell added a comment -

          Phil, could you review the change please?

          Show
          Robbie Gemmell added a comment - Phil, could you review the change please?
          Robbie Gemmell made changes -
          Status In Progress [ 3 ] Ready To Review [ 10006 ]
          Robbie Gemmell made changes -
          Status Reopened [ 4 ] In Progress [ 3 ]
          Hide
          Robbie Gemmell added a comment -

          I have applied a patch from Phil Harvey (found on QPID-4344) along with some additional changes of my own to resolve some issues (one caused by the prior changes in this JIRA) which could lead to the client not initiating failover when it should.

          Show
          Robbie Gemmell added a comment - I have applied a patch from Phil Harvey (found on QPID-4344 ) along with some additional changes of my own to resolve some issues (one caused by the prior changes in this JIRA) which could lead to the client not initiating failover when it should.
          Robbie Gemmell made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          Robbie Gemmell added a comment -

          Reopening due to issues discussed on QPID-4344 which resulted from the changes sin this JIRA.

          Show
          Robbie Gemmell added a comment - Reopening due to issues discussed on QPID-4344 which resulted from the changes sin this JIRA.
          Robbie Gemmell made changes -
          Link This issue is related to QPID-4344 [ QPID-4344 ]
          Robbie Gemmell made changes -
          Link This issue is related to QPID-3912 [ QPID-3912 ]
          Robbie Gemmell made changes -
          Status Ready To Review [ 10006 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Robbie Gemmell added a comment -

          Patch applied.

          Show
          Robbie Gemmell added a comment - Patch applied.
          Robbie Gemmell made changes -
          Summary [Java Client 0-8/0-9.x] Failover functionality does not restore the connection [Java Client 0-8/0-9.x] Failover functionality sometimes does not restore the connection
          Alex Rudyy made changes -
          Hide
          Alex Rudyy added a comment -

          Attached the final version of the patch

          Show
          Alex Rudyy added a comment - Attached the final version of the patch
          Alex Rudyy made changes -
          Attachment 0001-QPID-4289-Fix-failover-issues.patch [ 12544417 ]
          Alex Rudyy made changes -
          Attachment 0001-QPID-4289-Fix-failover-issues.patch [ 12544237 ]
          Alex Rudyy made changes -
          Attachment 0001-QPID-4289-Fix-failover-issues.patch [ 12544417 ]
          Alex Rudyy made changes -
          Attachment 0001-QPID-4289-Fix-failover-issues.patch [ 12544237 ]
          Alex Rudyy made changes -
          Attachment 0001-QPID-4289-Fix-failover-issues.patch [ 12544230 ]
          Alex Rudyy made changes -
          Attachment 0001-QPID-4289-Fix-failover-issues.patch [ 12544230 ]
          Alex Rudyy made changes -
          Attachment 0001-QPID-4289-Fix-failover-issues.patch [ 12544208 ]
          Alex Rudyy made changes -
          Description Failover functionality on 0-8/0-9.x java client does not restore the connection in the following scenarios:

          * when on initial establishing of the connectivity the connection object is marked as closed after to not being able to connect to the first brokers from the broker list but managed to connect to the one of the following brokers from the broker list
          * there are racing condition between waking up the connection state waiters and closing the phisical connection. As result, waiters are notified, new connection is created and closed afterwards.
          * on killing the broker when no IOException is thrown from socket input/output streams but -1 is returned by read operation instead, the failover starts but IoSender thread is left running. As result, on multiple reconnection attempts the thread resources can be exhausted and that can potentially result in OOME.
          Failover functionality on 0-8/0-9.x java client does not restore the connection in the following scenarios:

          * when on initial establishing of the connectivity the connection object is marked as closed after not being able to connect to the first brokers from the broker list but at the end it manages to connect to the one of the following brokers from the broker list
          * there is a racing condition between waking up the connection state waiter and closing the phisical connection on receiving ConnectionClose command from the broker. As result, waiters are notified, new connection is created and closed afterwards.
          * on killing the broker when no IOException is thrown from socket input/output streams but instead -1 is returned by read operation, the failover starts but IoSender thread is left running. As result, on multiple reconnection attempts the thread resources can be exhausted and that can potentially result in OOME.
          Alex Rudyy made changes -
          Assignee Alex Rudyy [ alex.rufous ] Robbie Gemmell [ gemmellr ]
          Hide
          Alex Rudyy added a comment -

          Robbie,
          Could you please review and commit the patch attached?

          Show
          Alex Rudyy added a comment - Robbie, Could you please review and commit the patch attached?
          Alex Rudyy made changes -
          Status In Progress [ 3 ] Ready To Review [ 10006 ]
          Alex Rudyy made changes -
          Attachment 0001-QPID-4289-Fix-failover-issues.patch [ 12544208 ]
          Hide
          Alex Rudyy added a comment -

          Attached a patch resolving the failover faunctionality issues on 0-9.x/0-8 path

          Show
          Alex Rudyy added a comment - Attached a patch resolving the failover faunctionality issues on 0-9.x/0-8 path
          Alex Rudyy made changes -
          Field Original Value New Value
          Status Open [ 1 ] In Progress [ 3 ]
          Alex Rudyy created issue -

            People

            • Assignee:
              Philip Harvey
              Reporter:
              Alex Rudyy
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development