[TINKERPOP-2541] .NET SDK (CosmosDb): The server returned status code '500' when status code '101' was expected.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 3.4.10
• Component/s: dotnet
• Flags: Important
• Assignee: Unassigned
• Reporter: Kirk Marple
• Created: 26/Mar/21 06:17
• Updated: 16/Sep/21 20:14
• Resolved: 16/Sep/21 20:14

Description

Using 3.4.10, .NET SDK, I've been getting a lot of problems talking to CosmosDb lately.

Some hangs on SubmitAsync (after several successful calls), and sometimes errors just creating the GremlinClient.

Seems to have cropped more over the last month, where I can reproduce it more often now. I've tried playing with pool size, etc. and haven't been able to get around it.

I'm able to repro it just from my laptop locally when running integration tests against CosmosDb.

Any thoughts on this? It's preventing us going into production with our application.
 

The server returned status code '500' when status code '101' was expected.
at System.Net.WebSockets.WebSocketHandle.<ConnectAsync>d__13.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Gremlin.Net.Driver.WebSocketConnection.<ConnectAsync>d__4.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Gremlin.Net.Driver.Connection.<ConnectAsync>d__15.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Gremlin.Net.Driver.ConnectionPool.<CreateNewConnectionAsync>d__18.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Gremlin.Net.Driver.ConnectionPool.<FillPoolAsync>d__17.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Gremlin.Net.Driver.ConnectionPool.<ReplaceDeadConnectionsAsync>d__15.MoveNext()
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Gremlin.Net.Process.Utils.WaitUnwrap(Task task)
at Gremlin.Net.Driver.ConnectionPool..ctor(IConnectionFactory connectionFactory, ConnectionPoolSettings settings)
at Gremlin.Net.Driver.GremlinClient..ctor(GremlinServer gremlinServer, GraphSONReader graphSONReader, GraphSONWriter graphSONWriter, String mimeType, ConnectionPoolSettings connectionPoolSettings, Action`1 webSocketConfiguration, String sessionId)
at Unstruk.Frameworks.Data.Graphs.GraphHelpers.<>c_DisplayClass4_0.<CreateGremlinClient>b_3() in D:\a\1\s\src\Unstruk.Frameworks.Data\GraphHelpers.cs:line 90
at Polly.Policy.<>c_DisplayClass126_0`1.<ExecuteAndCapture>b_0(Context ctx, CancellationToken ct)
at Polly.Retry.RetryEngine.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Action`4 onRetry, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider)
at Polly.Retry.RetryPolicy.Implementation[TResult](Func`3 action, Context context, CancellationToken cancellationToken)
at Polly.Policy.Execute[TResult](Func`3 action, Context context, CancellationToken cancellationToken)
at Polly.Policy.ExecuteAndCapture[TResult](Func`3 action, Context context, CancellationToken cancellationToken)

Here's my helper method for creating the client, which I wrapped with Polly retry, and it still failed.

        private const int DEFAULT_POOL_SIZE = 8;
        private const int DEFAULT_INPROCESS_PER_CONNECTION = 32;

        public static GremlinClient CreateGremlinClient(ILogger logger, GraphSettings settings, string collectionName)
        {
            const int DEFAULT_RETRY_COUNT = 5;
            const float RETRY_SEED = 1.5F;

            if (String.IsNullOrEmpty(settings.Location))
                throw new InvalidOperationException("Invalid CosmosDB graph server name.");

            if (String.IsNullOrEmpty(settings.Key))
                throw new InvalidOperationException("Invalid CosmosDB graph access key.");

            if (String.IsNullOrEmpty(settings.DatabaseId))
                throw new InvalidOperationException("Invalid CosmosDB graph database id.");

            if (String.IsNullOrEmpty(collectionName))
                throw new InvalidOperationException("Invalid CosmosDB graph collection name.");

            var sw = Stopwatch.StartNew();

            var uri = new Uri(settings.Location);

            uri = ConvertGremlinEndpoint(uri);

            var server = new GremlinServer(uri.Host, 443, true, $"/dbs/{settings.DatabaseId}/colls/{collectionName}", settings.Key);

            var connectionPoolSettings = new ConnectionPoolSettings
            {
                PoolSize = DEFAULT_POOL_SIZE,
                MaxInProcessPerConnection = DEFAULT_INPROCESS_PER_CONNECTION
            };

            var webSocketConfiguration = new Action<ClientWebSocketOptions>(options =>
            {
                //options.KeepAliveInterval = TimeSpan.FromSeconds(10);
            });

            int attempt = 1;

            var result = Policy
                .Handle<ResponseException>()
                .WaitAndRetry(DEFAULT_RETRY_COUNT,
                    retryAttempt => TimeSpan.FromSeconds(Math.Pow(RETRY_SEED, retryAttempt)),
                    onRetry: (e, delay) =>
                    {
#if DEBUG_SERVICE_GRAPH
                        logger.LogWarning($"Failed to create Gremlin client, attempt [{attempt}]. Trying again in [{delay}].  {e.Message}");
#endif

                        attempt++;
                    })
                .ExecuteAndCapture(() =>
                {
                    return new GremlinClient(server, new GraphSON2Reader(), new GraphSON2Writer(), GremlinClient.GraphSON2MimeType, connectionPoolSettings, webSocketConfiguration);
                });

            if (result.Outcome == OutcomeType.Failure)
                throw new InvalidOperationException($"Failed to create Gremlin Client.", result.FinalException);

            logger.LogDebug($"Created Gremlin client, location [{settings.Location}], database [{settings.DatabaseId}], collection [{collectionName}], took [{sw.Elapsed}].");

            return result.Result;
        }