Details
-
Bug
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
None
-
None
-
None
-
GCP
Description
I was trying to retrieve the LB info from my GCP project when I get the following error :
Traceback (most recent call last):
File "LbTestPy.py", line 41, in <module>
aLbs = lb_driver.list_balancers(ex_region="us-east1")
File "/home/cloud-user/LbTest/src/apache-libcloud/libcloud/loadbalancer/drivers/gce.py", line 87, in list_balancers
for fwr in self.gce.ex_list_forwarding_rules(region=ex_region):
File "/home/cloud-user/LbTest/src/apache-libcloud/libcloud/compute/drivers/gce.py", line 2092, in ex_list_forwarding_rules
for f in response['items']]
File "/home/cloud-user/LbTest/src/apache-libcloud/libcloud/compute/drivers/gce.py", line 7893, in _to_forwarding_rule
target = self._get_object_by_kind(forwarding_rule['target'])
File "/home/cloud-user/LbTest/src/apache-libcloud/libcloud/compute/drivers/gce.py", line 7249, in _get_object_by_kind
return GCENodeDriver.KIND_METHOD_MAP[response['kind']](self, response)
KeyError: 'compute#targetVpnGateway'
First I think it was my code/config but my test case is pretty simple (and I also lost the VM before the LB without issue).
After some investigation I manage to find the issue in libcloud/google.
The issue is that libcloud will list the forwarding rules to retrieve the info of the LoadBalancer. It retrieve the following forwarding rules :
https://cloud.google.com/compute/docs/load-balancing/network/forwarding-rules
In my case here is an extract of the HTTP response (retrieve by setting debug mode of libcloud):
- -------- begin 43335184 request ----------
curl -i -X GET -H 'Host: www.googleapis.com' -H 'Accept-Encoding: gzip,deflate' -H 'X-LC-Request-ID: 43335184' -H 'Content-Type: application/json' -H 'Authorization: Bearer ya29.El6eXXXXOc2Kn' -H 'User-Agent: libcloud/1.4.0 (Google Compute Engine) (Python 2.7.5/linux2)' --compress https://www.googleapis.com:443/compute/v1/projects/XXXXX/regions/us-east1/forwardingRules - -------- begin 43335184:43218488 response ----------
HTTP/1.1 200 OK
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Encoding: gzip
Transfer-Encoding: chunked
Expires: Tue, 22 Nov 2016 13:15:18 GMT
Vary: Origin, X-Origin
Server: GSE
Etag: "OKaT3lMknXXXXCJyDlI"
Cache-Control: private, max-age=0, must-revalidate, no-transform
Date: Tue, 22 Nov 2016 13:15:18 GMT
X-Frame-Options: SAMEORIGIN
Alt-Svc: quic=":443"; ma=2592000; v="36,35,34"
Content-Type: application/json; charset=UTF-8
444b
{
"kind": "compute#forwardingRuleList",
"id": "projects/XXXXX/regions/us-east1/forwardingRules",
"items": [
,
...
],
"selfLink": "https://www.googleapis.com/compute/v1/projects/XXXXX/regions/us-east1/forwardingRules"
}
The first thing that surprise me is that the response was big even if i have only 1 load balancer on this region. After it receive this response libcloud will try to convert each "forwading rule" from the message to a libcloud object with :
if 'items' in response: # The aggregated result returns dictionaries for each region if not global_rules and region is None: for v in response['items'].values(): region_forwarding_rules = [ self._to_forwarding_rule(f) for f in v.get('forwardingRules', []) ] list_forwarding_rules.extend(region_forwarding_rules) else: list_forwarding_rules = [self._to_forwarding_rule(f) for f in response['items']] return list_forwarding_rules
from def ex_list_forwarding_rules(self, region=None, global_rules=False):
To do so libcloud will call the following method "_to_forwarding_rule" on all items. This is where it break !
Indeed if you look on the answer of the "list forwardingRules" you will see that I have 2 types of rules :
Type 1 : The forward rules from the load balancer object :
{
"kind": "compute#forwardingRule",
"id": "6429XXXXXXXXXX887",
"creationTimestamp": "2016-11-21T09:46:32.011-08:00",
"name": "us-lb-forwarding-rule",
"description": "",
"region": "https://www.googleapis.com/compute/v1/projects/XXXXX/regions/us-east1",
"IPAddress": "1XXXXXXXXXX5",
"IPProtocol": "TCP",
"portRange": "30012-30012",
"target": "https://www.googleapis.com/compute/v1/projects/XXXXX/regions/us-east1/targetPools/us-lb",
"selfLink": "https://www.googleapis.com/compute/v1/projects/XXXXX/regions/us-east1/forwardingRules/us-lb-forwarding-rule",
"loadBalancingScheme": "EXTERNAL"
}
Type 2 : Forward rules from the VPN I have between projects on GCP
{
"kind": "compute#forwardingRule",
"id": "575XXXXXXXXXX282",
"creationTimestamp": "2016-07-11T01:31:17.574-07:00",
"name": "esp-acsgopstrain-a-us-east1",
"description": "",
"region": "https://www.googleapis.com/compute/v1/projects/XXXXX/regions/us-east1",
"IPAddress": "XXXXXXXXXX",
"IPProtocol": "ESP",
"target": "https://www.googleapis.com/compute/v1/projects/XXXXX/regions/us-east1/targetVpnGateways/acsgopstrain-a-us-east1",
"selfLink": "https://www.googleapis.com/compute/v1/projects/XXXXX/regions/us-east1/forwardingRules/esp-acsgopstrain-a-us-east1",
"loadBalancingScheme": "EXTERNAL"
},
The libcloud method "_to_forwarding_rule" will works fine on the type 1. Here is the code :
def _to_forwarding_rule(self, forwarding_rule):
"""
Return a Forwarding Rule object from the JSON-response dictionary.
:param forwarding_rule: The dictionary describing the rule.
:type forwarding_rule: ``dict``
:return: ForwardingRule object
:rtype: :class:`GCEForwardingRule`
"""
extra = {}
extra['selfLink'] = forwarding_rule.get('selfLink')
extra['portRange'] = forwarding_rule.get('portRange')
extra['creationTimestamp'] = forwarding_rule.get('creationTimestamp')
extra['description'] = forwarding_rule.get('description')
region = forwarding_rule.get('region')
if region:
region = self.ex_get_region(region)
target = self._get_object_by_kind(forwarding_rule['target'])
The "_get_object_by_kind" will works fine because the target of the forward rule for type 1 is "targetpool" as you can see :
"https://www.googleapis.com/compute/v1/projects/XXXXX/regions/us-east1/targetPools/us-lb",
whereas it will crash with the stack i show you before for the type 2 because the target is "targetVpnGateways" as you can see :
"https://www.googleapis.com/compute/v1/projects/XXXXX/regions/us-east1/targetVpnGateways/acsgopstrain-a-us-east1",
I think the original implementation of the LB on libcloud for GCE did not know that Google will mix several rules for both the LoadBalancer AND the VPN.
I made a dirty fix on my libcloud master code to only try to convert the forward rules link to a LB :
else: list_forwarding_rules = [self._to_forwarding_rule(f) for f in response['items'] if (not "targetVpnGateways" in f['target'])]
This fix the issue !
To be honest I m not sure what to do....I would except Google to not mix the forward rules of VPN and LB. There is nothing mentioning the VPN in the forwarding rules in the google doc (https://cloud.google.com/compute/docs/load-balancing/network/forwarding-rules) and there is nothing about forwarding rule in the VPN documentation (https://cloud.google.com/compute/docs/vpn/networks)
I would suggest to do a temporary hack (similar in spirit of the one i done) until we clarify that with Google.