Uploaded image for project: 'jclouds'
  1. jclouds
  2. JCLOUDS-1294

Azure ARM improve HTTP retry logic

    XMLWordPrintableJSON

Details

    Description

      The retry logic in AzureRateLimitRetryHandler doesn't catch all requests that can be retried.

      >> PUT https://management.azure.com/subscriptions/<subscription id>/resourcegroups/jclouds-westeurope/providers/Microsoft.Network/virtualNetworks/network-security-vnet-q8nvtq9kkr/subnets/web-tier?api-version=2017-03-01 HTTP/1.1
      >> Accept: application/json
      >> Content-Type: application/json
      >> Content-Length: 48
      ...
      2017-05-12 13:40:31,962 DEBUG 106 o.j.i.ContentMetadataCodec$DefaultContentMetadataCodec [ager-N7FStlcv-20] Invalid Expires header (-1); should be in RFC-1123 format; treating as already expired: Error parsing data at 0
      2017-05-12 13:40:31,962 DEBUG 106 o.j.h.o.OkHttpCommandExecutorService [ager-N7FStlcv-20] Receiving response 290079285: HTTP/1.1 429 
      << HTTP/1.1 429 
      << Connection: close
      << Date: Fri, 12 May 2017 10:40:30 GMT
      << OkHttp-Received-Millis: 1494585631962
      << OkHttp-Selected-Protocol: http/1.1
      << OkHttp-Sent-Millis: 1494585586349
      << Pragma: no-cache
      << Server: Microsoft-HTTPAPI/2.0
      << Strict-Transport-Security: max-age=31536000; includeSubDomains
      << x-ms-correlation-request-id: 4b7a44ea-1976-4dc2-95ef-1a7c80f82d3c
      << x-ms-ratelimit-remaining-subscription-writes: 1195
      << x-ms-request-id: 4b7a44ea-1976-4dc2-95ef-1a7c80f82d3c
      << x-ms-routing-request-id: CANADAEAST:20170512T104031Z:4b7a44ea-
      << Cache-Control: no-cache
      << Content-Type: application/json; charset=utf-8
      << Content-Length: 2545
      << Expires: Thu Jan 01 02:00:00 EET 1970
      2017-05-12 13:40:31,963 ERROR 106 o.j.a.a.h.AzureRateLimitRetryHandler [ager-N7FStlcv-20] Cannot retry after rate limit error, no retry information provided in the response
      ...
      2017-05-12 13:40:31,967 DEBUG 126 o.a.b.l.j.JcloudsLocation [ager-N7FStlcv-20] org.jclouds.azurecompute.arm.exceptions.AzureComputeRateLimitExceededException: HTTP/1.1 429 
      {x-ms-ratelimit-remaining-subscription-writes=[1195]}
      ...
      Caused by: org.jclouds.http.HttpResponseException: command: PUT https://management.azure.com/subscriptions/<subscription id>/resourcegroups/jclouds-westeurope/providers/Microsoft.Network/virtualNetworks/network-security-vnet-q8nvtq9kkr/subnets/web-tier HTTP/1.1 failed with response: HTTP/1.1 429 ; content: [{"Error":{"Details":[{"Code":"GatewayError","Message":"Error occured in resource provider infrastructure services.","Target":null}],"InnerError":"Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.NrpException: A retryable error occured. ---> Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.NrpException: Error occured in resource provider infrastructure services. ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n   at Microsoft.WindowsAzure.Networking.Nrp.Common.ReliableHttpClient.HttpClientWrapper.<SendAsync>d__4.MoveNext() in E:\\bt\\804732\\repo\\src\\sources\\Common\\ReliableHttpClient\\HttpClientWrapper.cs:line 80\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n   at Microsoft.WindowsAzure.Networking.Nrp.GatewayServiceCommon.GatewayServiceRoutingController.<CallNrpAndGetResponseWithRetries>d__23.MoveNext() in E:\\bt\\804732\\repo\\src\\sources\\GatewayService\\GatewayCommon\\GatewayServiceRoutingController.cs:line 414\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()\r\n   at Microsoft.WindowsAzure.Networking.Nrp.GatewayServiceCommon.GatewayServiceRoutingController.<ForwardMessageToNrpAndProcessResponseAsync>d__20.MoveNext() in E:\\bt\\804732\\repo\\src\\sources\\GatewayService\\GatewayCommon\\GatewayServiceRoutingController.cs:line 160\r\n   --- End of inner exception stack trace ---\r\n   --- End of inner exception stack trace ---","Code":"RetryableError","Message":"A retryable error occured.","Target":null}}]
              at org.jclouds.azurecompute.arm.handlers.AzureComputeErrorHandler.handleError(AzureComputeErrorHandler.java:45)
              ... 33 more
      
      

      The response body expanded here for readability:

      PUT https://management.azure.com/subscriptions/341751b0-f348-45ce-9498-41cc68b4b45f/resourcegroups/jclouds-westeurope/providers/Microsoft.Network/virtualNetworks/network-security-vnet-q8nvtq9kkr/subnets/web-tier HTTP/1.1 
      failed with response: HTTP/1.1 429 ; content: 
      [{"Error":{"Details":[{"Code":"GatewayError","Message":"Error occured in resource provider infrastructure services.","Target":null}],
      "InnerError":"Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.NrpException: A retryable error occured. 
      ---> Microsoft.WindowsAzure.Networking.Nrp.Frontend.Common.NrpException: Error occured in resource provider infrastructure services. 
      ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
         at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
         at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
         at Microsoft.WindowsAzure.Networking.Nrp.Common.ReliableHttpClient.HttpClientWrapper.<SendAsync>d__4.MoveNext() in E:\\bt\\804732\\repo\\src\\sources\\Common\\ReliableHttpClient\\HttpClientWrapper.cs:line 80
      --- End of stack trace from previous location where exception was thrown ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
         at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
         at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
         at Microsoft.WindowsAzure.Networking.Nrp.GatewayServiceCommon.GatewayServiceRoutingController.<CallNrpAndGetResponseWithRetries>d__23.MoveNext() in E:\\bt\\804732\\repo\\src\\sources\\GatewayService\\GatewayCommon\\GatewayServiceRoutingController.cs:line 414
      --- End of stack trace from previous location where exception was thrown ---
         at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
         at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
         at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
         at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
         at Microsoft.WindowsAzure.Networking.Nrp.GatewayServiceCommon.GatewayServiceRoutingController.<ForwardMessageToNrpAndProcessResponseAsync>d__20.MoveNext() in E:\\bt\\804732\\repo\\src\\sources\\GatewayService\\GatewayCommon\\GatewayServiceRoutingController.cs:line 160
         --- End of inner exception stack trace ---
         --- End of inner exception stack trace ---
      ","Code":"RetryableError","Message":"A retryable error occured.","Target":null}}]
      
      

      Not all retriable requests get the Retry-After header.

      Attachments

        Activity

          People

            Unassigned Unassigned
            svet Svetoslav Neykov
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: