Uploaded image for project: 'TinkerPop'
  1. TinkerPop
  2. TINKERPOP-2030

KeepAlive task executed for every Connection.write call

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.3.2
    • Fix Version/s: 3.4.0, 3.3.4, 3.2.10
    • Component/s: driver
    • Labels:
      None

      Description

      The Gremlin Driver communicates with the server via Connection.write().

      The write() method has a logic in the end to schedule a task to keep the connection alive, which by default is scheduled to be run after 30min, every 30 min.

        // try to keep the connection alive if the channel allows such things - websockets will
              if (channelizer.supportsKeepAlive() && keepAliveInterval > 0) {
      
                  final ScheduledFuture oldKeepAliveFuture = keepAliveFuture.getAndSet(cluster.executor().scheduleAtFixedRate(() -> {
                      logger.debug("Request sent to server to keep {} alive", thisConnection);
                      try {
                          channel.writeAndFlush(channelizer.createKeepAliveMessage());
                      } catch (Exception ex) {
                          // will just log this for now - a future real request can be responsible for the failure that
                          // marks the host as dead. this also may not mean the host is actually dead. more robust handling
                          // is in play for real requests, not this simple ping
                          logger.warn(String.format("Keep-alive did not succeed on %s", thisConnection), ex);
                      }
                  }, keepAliveInterval, keepAliveInterval, TimeUnit.MILLISECONDS));
      
                  // try to cancel the old future if it's still un-executed - no need to ping since a new write has come
                  // through on the connection
                  if (oldKeepAliveFuture != null) oldKeepAliveFuture.cancel(true);
              }
      

      The problem with this is that on every call of the wirte() method, which basically means on every query executed, the {[Connection}} schedule a KeepAlive task which won't be run for at least 30 minutes. This lead the Cluster.executor() queue to fill up with tasks waiting for completion.

      One possible solution to fix this would be to schedule avoid this KeepAlive task to be instantiated more than once per connection:

      final class Connection {
          ...
          private final AtomicBoolean keepAliveInitialized = new AtomicBoolean(false);
      
          public ChannelPromise write(final RequestMessage requestMessage, final CompletableFuture<ResultSet> future) {
              ...
              
              // FIX HERE: with keepAliveInitialized.compareAndSet(false, true) we ensure the keepAlive task is run only once per connection.
              if (channelizer.supportsKeepAlive() && keepAliveInterval > 0 && keepAliveInitialized.compareAndSet(false, true)) {
                  ...
              }
          }
      }
      

        Attachments

          Activity

            People

            • Assignee:
              spmallette stephen mallette
              Reporter:
              srondelli Simone Rondelli
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: