Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Compute
    • Labels:
      None
    • Environment:

      apache-libcloud==0.18.0
      Ubuntu 14.04
      Python2.7

      Description

      When attempting to connect to Linode through the Linode provider, I am receiving a dropped connection. The following DEBUG information is attached.

      I have also found that when taking the below curl request and sending it normally, I do not have any errors on getting a response.

      1. -------- begin 140040111659984 request ----------
        curl -i -X GET -H 'Host: api.linode.com' -H 'X-LC-Request-ID: 140040111659984' -H 'Accept-Encoding: gzip,deflate' -H 'User-Agent: libcloud/0.18.0 (Linode) ' --compress 'https://api.linode.com:443/?api_key=sekrit&api_responseFormat=json&api_action=avail.linodeplans'
        Traceback (most recent call last):
        File "test.py", line 6, in <module>
        p.worker.consume_queue()
        File "/srv/auction-balancer/Auction-Balancer/sae/prov/util/worker.py", line 53, in wrapped_func
        channel.start_consuming()
        File "/srv/auction-balancer/Auction-Balancer/provenv/local/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 1681, in start_consuming
        self.connection.process_data_events(time_limit=None)
        File "/srv/auction-balancer/Auction-Balancer/provenv/local/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 656, in process_data_events
        self._dispatch_channel_events()
        File "/srv/auction-balancer/Auction-Balancer/provenv/local/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 469, in _dispatch_channel_events
        impl_channel._get_cookie()._dispatch_events()
        File "/srv/auction-balancer/Auction-Balancer/provenv/local/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 1310, in _dispatch_events
        evt.body)
        File "/srv/auction-balancer/Auction-Balancer/sae/prov/util/worker.py", line 45, in callback
        success = func(message)
        File "/srv/auction-balancer/Auction-Balancer/sae/prov/util/worker.py", line 102, in consume_queue
        node = server.create_node('linode', name=machine_name, size_id='Linode 1024', image_id='Ubuntu 14.04 LTS')
        File "/srv/auction-balancer/Auction-Balancer/sae/prov/util/provider.py", line 78, in create_node
        size = [s for s in driver.list_sizes() if s.name == size_id][0]
        File "/srv/auction-balancer/Auction-Balancer/provenv/local/lib/python2.7/site-packages/libcloud/compute/drivers/linode.py", line 405, in list_sizes
        data = self.connection.request(API_ROOT, params=params).objects[0]
        File "/srv/auction-balancer/Auction-Balancer/provenv/local/lib/python2.7/site-packages/libcloud/common/base.py", line 784, in request
        headers=headers)
        File "/srv/auction-balancer/Auction-Balancer/provenv/local/lib/python2.7/site-packages/libcloud/common/base.py", line 443, in request
        headers)
        File "/usr/lib/python2.7/httplib.py", line 979, in request
        self._send_request(method, url, body, headers)
        File "/usr/lib/python2.7/httplib.py", line 1013, in _send_request
        self.endheaders(body)
        File "/usr/lib/python2.7/httplib.py", line 975, in endheaders
        self._send_output(message_body)
        File "/usr/lib/python2.7/httplib.py", line 835, in _send_output
        self.send(msg)
        File "/usr/lib/python2.7/httplib.py", line 797, in send
        self.connect()
        File "/srv/auction-balancer/Auction-Balancer/provenv/local/lib/python2.7/site-packages/libcloud/httplib_ssl.py", line 280, in connect
        ssl_version=libcloud.security.SSL_VERSION)
        File "/usr/lib/python2.7/ssl.py", line 487, in wrap_socket
        ciphers=ciphers)
        File "/usr/lib/python2.7/ssl.py", line 243, in _init_
        self.do_handshake()
        File "/usr/lib/python2.7/ssl.py", line 405, in do_handshake
        self._sslobj.do_handshake()
        socket.error: [Errno 104] Connection reset by peer

      Testing with a separate library also based on httplib , I found that it is not failing when connecting to the Linode API however.

        Activity

        Hide
        kami Tomaz Muraus added a comment -

        Thanks for the report.

        The DEUBG log you pasted seems to correspond to the pika library and not libcloud so it's not helpful here.

        It would help us if you include `LIBCLOUD_DEBUG` output as described here - https://libcloud.readthedocs.org/en/latest/troubleshooting.html?highlight=libcloud_debug#debugging

        Show
        kami Tomaz Muraus added a comment - Thanks for the report. The DEUBG log you pasted seems to correspond to the pika library and not libcloud so it's not helpful here. It would help us if you include `LIBCLOUD_DEBUG` output as described here - https://libcloud.readthedocs.org/en/latest/troubleshooting.html?highlight=libcloud_debug#debugging
        Hide
        jacob-sae Jacob Riley added a comment -

        Hello Tomaz,

        Thank you for getting back to me. I'm terribly new to development so I really appreciate your input.

        I've changed the DEBUG output to remove all pika entries and to only include relevant debug information.

        Let me know if there's anything else I can do to help.

        Show
        jacob-sae Jacob Riley added a comment - Hello Tomaz, Thank you for getting back to me. I'm terribly new to development so I really appreciate your input. I've changed the DEBUG output to remove all pika entries and to only include relevant debug information. Let me know if there's anything else I can do to help.
        Hide
        kami Tomaz Muraus added a comment -

        Hm, some other users also encountered a similar issue in the past.

        Sadly, we have never been able to track down a root cause (it seems like a networking issue and not an actual issue in Libcloud - the reason you can't replicate it with curl is that it's a temporary / intermediate network issue). The good news is that we have a work-around for it in trunk (it will be available in the next release).

        This work-around allows libcloud to automatically retry timed out or failed HTTP requests - https://github.com/apache/libcloud/pull/556

        Show
        kami Tomaz Muraus added a comment - Hm, some other users also encountered a similar issue in the past. Sadly, we have never been able to track down a root cause (it seems like a networking issue and not an actual issue in Libcloud - the reason you can't replicate it with curl is that it's a temporary / intermediate network issue). The good news is that we have a work-around for it in trunk (it will be available in the next release). This work-around allows libcloud to automatically retry timed out or failed HTTP requests - https://github.com/apache/libcloud/pull/556
        Hide
        jacob-sae Jacob Riley added a comment -

        Hello Tomaz,

        I have also found that recently Linode made an update to their API that blocks TLS v1.0 and below. My server has ciphers that support TLS1.1+ , so this wouldn't be the problem correct?

        I'm taking my software and installing on a completely separate instance to see if it still works.

        Again, thank you so much for your attention to this and if there's anything further you think I could try as a test, I'd be 100% up for attempting it.

        Show
        jacob-sae Jacob Riley added a comment - Hello Tomaz, I have also found that recently Linode made an update to their API that blocks TLS v1.0 and below. My server has ciphers that support TLS1.1+ , so this wouldn't be the problem correct? I'm taking my software and installing on a completely separate instance to see if it still works. Again, thank you so much for your attention to this and if there's anything further you think I could try as a test, I'd be 100% up for attempting it.
        Hide
        kami Tomaz Muraus added a comment -

        Does this error happens always or only sometimes?

        If it happens always, then supported TLS version and ciphers could have something to do with it.

        Show
        kami Tomaz Muraus added a comment - Does this error happens always or only sometimes? If it happens always, then supported TLS version and ciphers could have something to do with it.
        Hide
        jacob-sae Jacob Riley added a comment -

        This error has happened consistently on every attempt for the last two weeks since the change made by Linode to mitigate incoming DDoS attacks.

        Show
        jacob-sae Jacob Riley added a comment - This error has happened consistently on every attempt for the last two weeks since the change made by Linode to mitigate incoming DDoS attacks.
        Hide
        kami Tomaz Muraus added a comment -

        In this case it could be related to the used TLS version and / or ciphers.

        It could also be that the Linode DDoS mitigation is inadvertently dropping your requests made through Libcloud.

        I would recommend you to contact Linode and provide them with this information. Maybe they might have a clue at what might be going on.

        Show
        kami Tomaz Muraus added a comment - In this case it could be related to the used TLS version and / or ciphers. It could also be that the Linode DDoS mitigation is inadvertently dropping your requests made through Libcloud. I would recommend you to contact Linode and provide them with this information. Maybe they might have a clue at what might be going on.
        Hide
        stevednd Steve V added a comment -

        I'm experiencing the same issue using libcloud through saltstack. I can interact with the Linode API through the command line using curl no problem from the same machine though. So it seems that with what Jacob is also saying, something about the way that libcloud calls the API is causing the failure.

        I just heard from Linode, and they are saying TLS v1.2 is required. The server I am trying from indeed supports TLS 1.2, but maybe libcloud is attempting to use the wrong version?

        Show
        stevednd Steve V added a comment - I'm experiencing the same issue using libcloud through saltstack. I can interact with the Linode API through the command line using curl no problem from the same machine though. So it seems that with what Jacob is also saying, something about the way that libcloud calls the API is causing the failure. I just heard from Linode, and they are saying TLS v1.2 is required. The server I am trying from indeed supports TLS 1.2, but maybe libcloud is attempting to use the wrong version?
        Hide
        jacob-sae Jacob Riley added a comment -

        It would make sense.

        curl -i -X GET -H 'Host: api.linode.com' -H 'X-LC-Request-ID: 140040111659984' -H 'Accept-Encoding: gzip,deflate' -H 'User-Agent: libcloud/0.18.0 (Linode) ' --compress 'https://api.linode.com:443/?api_key=sekrit&api_responseFormat=json&api_action=avail.linodeplans'

        This curl request works as long as I can force the TLS version to be 1.2 when connecting.

        Is it possible that the curl request internal to libcloud is not the same as the one shown in the DEBUG output?

        I'll do some more searching and it turns out the ticket I had with Linode going for the last week was closed.

        Show
        jacob-sae Jacob Riley added a comment - It would make sense. curl -i -X GET -H 'Host: api.linode.com' -H 'X-LC-Request-ID: 140040111659984' -H 'Accept-Encoding: gzip,deflate' -H 'User-Agent: libcloud/0.18.0 (Linode) ' --compress 'https://api.linode.com:443/?api_key=sekrit&api_responseFormat=json&api_action=avail.linodeplans' This curl request works as long as I can force the TLS version to be 1.2 when connecting. Is it possible that the curl request internal to libcloud is not the same as the one shown in the DEBUG output? I'll do some more searching and it turns out the ticket I had with Linode going for the last week was closed.
        Hide
        Heath_N Heath Naylor added a comment - - edited

        Looks to be using TLS v1.0? Is this something we need to get changed?

        https://github.com/apache/libcloud/blob/ecbaa0b57b1b1fcbda11cef60bcaae8ee67a9340/libcloud/security.py#L37

        Python ssl library, 1.1 and 1.2 requires python version 2.7.9
        https://docs.python.org/2/library/ssl.html#ssl.PROTOCOL_TLSv1_2

        Show
        Heath_N Heath Naylor added a comment - - edited Looks to be using TLS v1.0? Is this something we need to get changed? https://github.com/apache/libcloud/blob/ecbaa0b57b1b1fcbda11cef60bcaae8ee67a9340/libcloud/security.py#L37 Python ssl library, 1.1 and 1.2 requires python version 2.7.9 https://docs.python.org/2/library/ssl.html#ssl.PROTOCOL_TLSv1_2
        Hide
        jacob-sae Jacob Riley added a comment -

        Linode doesn't support TLS1.0 any longer, they changed to support TLS1.1 and greater after the DDoS they experienced recently.

        Show
        jacob-sae Jacob Riley added a comment - Linode doesn't support TLS1.0 any longer, they changed to support TLS1.1 and greater after the DDoS they experienced recently.
        Hide
        Heath_N Heath Naylor added a comment -

        Jacob, understood, this means we need to get libcloud to update to PROTOCOL_TLSv1_1 or PROTOCOL_TLSv1_2. Also need to ensure that we have python 2.7.9 updated.

        Show
        Heath_N Heath Naylor added a comment - Jacob, understood, this means we need to get libcloud to update to PROTOCOL_TLSv1_1 or PROTOCOL_TLSv1_2. Also need to ensure that we have python 2.7.9 updated.
        Hide
        kami Tomaz Muraus added a comment -

        Good catch everyone.

        I believe we set that to TLS v1.0 for backward compatibility reasons since we also support older versions if Python.

        For one, I'm OK with defaulting to a newer version (PROTOCOL_TLSv1_1 or PROTOCOL_TLSv1_2) if it's available on that system.

        Having said that, the option itself is already configurable, you can do:

        import libcloud.security
        libcloud.security.SSL_VERSION = ssl.PROTOCOL_TLSv1_1
        

        Keep in mind that you need to do at the beginning, before any other Libcloud imports.

        I will also make sure this gets documented. Thanks again.

        Show
        kami Tomaz Muraus added a comment - Good catch everyone. I believe we set that to TLS v1.0 for backward compatibility reasons since we also support older versions if Python. For one, I'm OK with defaulting to a newer version (PROTOCOL_TLSv1_1 or PROTOCOL_TLSv1_2) if it's available on that system. Having said that, the option itself is already configurable, you can do: import libcloud.security libcloud.security.SSL_VERSION = ssl.PROTOCOL_TLSv1_1 Keep in mind that you need to do at the beginning, before any other Libcloud imports. I will also make sure this gets documented. Thanks again.
        Hide
        kami Tomaz Muraus added a comment -

        I wrote a short announcement on how to work around this issue - https://libcloud.apache.org/blog/2016/01/14/notice-for-linode-users.html

        I will also look if it's possible to default to a higher (and more secure) version. Sadly I don't think this will make a much of a difference, since afaik, not many people actually use Python 2.7.9 or Python >= 3.4 yet.

        Show
        kami Tomaz Muraus added a comment - I wrote a short announcement on how to work around this issue - https://libcloud.apache.org/blog/2016/01/14/notice-for-linode-users.html I will also look if it's possible to default to a higher (and more secure) version. Sadly I don't think this will make a much of a difference, since afaik, not many people actually use Python 2.7.9 or Python >= 3.4 yet.
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user Kami opened a pull request:

        https://github.com/apache/libcloud/pull/682

        Throw a more friendly error message if establishing SSL / TLS connection fails

        With this change we now throw a more user-friendly error message in case establishing SSL / TLS connection fails.

        New error message:

        ```python
        File "/home/kami/w/lc/libcloud/libcloud/httplib_ssl.py", line 312, in connect
        raise new_e
        socket.error: Failed to establish SSL / TLS connection ([Errno 104] Connection reset by peer). It is possible that the server doesn't support requested SSL / TLS version (TLS v1.0).
        For information on how to work around this issue, please see https://libcloud.readthedocs.org/en/latest/other/ssl-certificate-validation.html#changing-used-ssl-tls-version
        ```

        Related to https://issues.apache.org/jira/browse/LIBCLOUD-791

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/Kami/libcloud friendlier_error_message_on_tls_error

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/libcloud/pull/682.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #682


        commit a9acb7fca02867c3d1aa101c9d457803cfb7aef1
        Author: Tomaz Muraus <tomaz@apache.org>
        Date: 2016-01-14T12:47:09Z

        Throw a more friendly error message if establishing SSL / TLS connection fails.


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user Kami opened a pull request: https://github.com/apache/libcloud/pull/682 Throw a more friendly error message if establishing SSL / TLS connection fails With this change we now throw a more user-friendly error message in case establishing SSL / TLS connection fails. New error message: ```python File "/home/kami/w/lc/libcloud/libcloud/httplib_ssl.py", line 312, in connect raise new_e socket.error: Failed to establish SSL / TLS connection ( [Errno 104] Connection reset by peer). It is possible that the server doesn't support requested SSL / TLS version (TLS v1.0). For information on how to work around this issue, please see https://libcloud.readthedocs.org/en/latest/other/ssl-certificate-validation.html#changing-used-ssl-tls-version ``` Related to https://issues.apache.org/jira/browse/LIBCLOUD-791 You can merge this pull request into a Git repository by running: $ git pull https://github.com/Kami/libcloud friendlier_error_message_on_tls_error Alternatively you can review and apply these changes as the patch at: https://github.com/apache/libcloud/pull/682.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #682 commit a9acb7fca02867c3d1aa101c9d457803cfb7aef1 Author: Tomaz Muraus <tomaz@apache.org> Date: 2016-01-14T12:47:09Z Throw a more friendly error message if establishing SSL / TLS connection fails.
        Hide
        kami Tomaz Muraus added a comment -

        To clarify it further - another option is to use ssl.PROTOCOL_SSLv23 constants which will let the client and server negotiate the highest supported protocol version (>= SSL v3.0 <= TLS v1.2). That's also the default option used by Python.

        The reason why we explicitly chose TLS v1.0 and not this constant is that it's more secure. Using ssl.PROTOCOL_SSLv23 exposes user to potential downgrade attacks to SSL v3.0 and SSL v3.0 is considered broken and unsafe.

        I added some clarifications about that to our documentation - https://github.com/apache/libcloud/commit/4bb534c063b222f94fe56c22e54345826280cbae

        In addition to that, I opened a pull request so we will now throw a more user-friendly error message if this conditions occurs - https://github.com/apache/libcloud/pull/682

        Show
        kami Tomaz Muraus added a comment - To clarify it further - another option is to use ssl.PROTOCOL_SSLv23 constants which will let the client and server negotiate the highest supported protocol version (>= SSL v3.0 <= TLS v1.2). That's also the default option used by Python. The reason why we explicitly chose TLS v1.0 and not this constant is that it's more secure. Using ssl.PROTOCOL_SSLv23 exposes user to potential downgrade attacks to SSL v3.0 and SSL v3.0 is considered broken and unsafe. I added some clarifications about that to our documentation - https://github.com/apache/libcloud/commit/4bb534c063b222f94fe56c22e54345826280cbae In addition to that, I opened a pull request so we will now throw a more user-friendly error message if this conditions occurs - https://github.com/apache/libcloud/pull/682
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/libcloud/pull/682

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/libcloud/pull/682

          People

          • Assignee:
            Unassigned
            Reporter:
            jacob-sae Jacob Riley
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development