Uploaded image for project: 'TinkerPop'
  1. TinkerPop
  2. TINKERPOP-1249

Gremlin driver to periodically issue ping / heartbeat to gremlin server

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Implemented
    • Affects Version/s: 3.1.1-incubating
    • Fix Version/s: 3.2.3
    • Component/s: driver
    • Labels:
      None
    • Environment:
      gremlin driver accessing gremlin server behind a load balancer / proxy (tested on nginx proxing the gremlin server)

      Description

      Gremlin driver currently not sending any ping request to the gremlin server. As a result, the websocket channel gets closed by the nginx / load balancer/ proxy after a period of inactivity.

      This forces the sockets to be left open indefinitely on the proxy/ load balancer. Which is not a good practice, because this would lead to socket leaks (not sure if that is the right word). What I mean by saying socket leak, is: if there are multiple clients (micro services) connecting to the same gremlin-server using gremlin driver. They all end up having open channels to the server. And the server would never be able to release the channels.

      Meaning if the micro services using gremlin driver and having a pool size of 10 are restarted say 100 times, there will be 1000 channels open indefinitely and there is no way to revoke the killed instances' open channels.

      This has been discussed in gremlin-users group: https://groups.google.com/forum/#!topic/gremlin-users/UjaV6sRuKMc

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/tinkerpop/pull/433

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tinkerpop/pull/433
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user PommeVerte commented on the issue:

          https://github.com/apache/tinkerpop/pull/433

          VOTE +1 I get the pinging as well. Looks good.

          Show
          githubbot ASF GitHub Bot added a comment - Github user PommeVerte commented on the issue: https://github.com/apache/tinkerpop/pull/433 VOTE +1 I get the pinging as well. Looks good.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user dkuppitz commented on the issue:

          https://github.com/apache/tinkerpop/pull/433

          `docker/build.sh -t -i -n` succeeded.

          VOTE: +1

          Show
          githubbot ASF GitHub Bot added a comment - Github user dkuppitz commented on the issue: https://github.com/apache/tinkerpop/pull/433 `docker/build.sh -t -i -n` succeeded. VOTE: +1
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user robertdale commented on the issue:

          https://github.com/apache/tinkerpop/pull/433

          Yup, there it is right there in the javadoc: When close() is called, the Cluster is left open for the caller to close.

          Show
          githubbot ASF GitHub Bot added a comment - Github user robertdale commented on the issue: https://github.com/apache/tinkerpop/pull/433 Yup, there it is right there in the javadoc: When close() is called, the Cluster is left open for the caller to close.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user spmallette commented on the issue:

          https://github.com/apache/tinkerpop/pull/433

          @robertdale thanks for testing that. If you pass in your own `Cluster`, then the `DriverRemoteConnection`, expects you to manage it. In this way, a `Cluster` can be shared.

          Show
          githubbot ASF GitHub Bot added a comment - Github user spmallette commented on the issue: https://github.com/apache/tinkerpop/pull/433 @robertdale thanks for testing that. If you pass in your own `Cluster`, then the `DriverRemoteConnection`, expects you to manage it. In this way, a `Cluster` can be shared.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user robertdale commented on the issue:

          https://github.com/apache/tinkerpop/pull/433

          Use your favorite packet sniffer. I used tcpdump.

          Show
          githubbot ASF GitHub Bot added a comment - Github user robertdale commented on the issue: https://github.com/apache/tinkerpop/pull/433 Use your favorite packet sniffer. I used tcpdump.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user k4rthikr commented on the issue:

          https://github.com/apache/tinkerpop/pull/433

          How do I know that pings are happening in the background? In other words, how do I test this?

          Show
          githubbot ASF GitHub Bot added a comment - Github user k4rthikr commented on the issue: https://github.com/apache/tinkerpop/pull/433 How do I know that pings are happening in the background? In other words, how do I test this?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user robertdale commented on the issue:

          https://github.com/apache/tinkerpop/pull/433

          Tested a clean console build with 3.2.2 server. Pinging works as expected. Closing each Client, a Cluster, and ':remote close' does stop pinging as expected.

          Not sure what the expected behavior is here but this does not stop pinging after close. I had expected it to work like Client. Instead the Cluster must be closed.
          ```
          cluster = Cluster.open('conf/remote-objects.yaml')
          rem = DriverRemoteConnection.using(cluster, 'g')
          g = EmptyGraph.instance().traversal().withRemote(rem)
          g.V()
          rem.close()
          ```

          Show
          githubbot ASF GitHub Bot added a comment - Github user robertdale commented on the issue: https://github.com/apache/tinkerpop/pull/433 Tested a clean console build with 3.2.2 server. Pinging works as expected. Closing each Client, a Cluster, and ':remote close' does stop pinging as expected. Not sure what the expected behavior is here but this does not stop pinging after close. I had expected it to work like Client. Instead the Cluster must be closed. ``` cluster = Cluster.open('conf/remote-objects.yaml') rem = DriverRemoteConnection.using(cluster, 'g') g = EmptyGraph.instance().traversal().withRemote(rem) g.V() rem.close() ```
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user spmallette opened a pull request:

          https://github.com/apache/tinkerpop/pull/433

          TINKERPOP-1249 Add keep-alive functionality to websockets Java Driver

          https://issues.apache.org/jira/browse/TINKERPOP-1249

          If a connection goes idle, the driver will periodically send a ping to the server to keep the connection alive.

          All good with `mvn clean install && mvn verify -pl gremlin-server -DskipIntegrationTests=false`. Also ran some manual tests and did some memory profiling of the driver to make sure there were no leaks.

          VOTE +1

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/apache/tinkerpop TINKERPOP-1249

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tinkerpop/pull/433.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #433


          commit d881484a40ef7c5924e97a1adce7d0a7bf6654ea
          Author: Stephen Mallette <spmva@genoprime.com>
          Date: 2016-09-22T11:48:59Z

          Add keep-alive functionality to Java Driver.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user spmallette opened a pull request: https://github.com/apache/tinkerpop/pull/433 TINKERPOP-1249 Add keep-alive functionality to websockets Java Driver https://issues.apache.org/jira/browse/TINKERPOP-1249 If a connection goes idle, the driver will periodically send a ping to the server to keep the connection alive. All good with `mvn clean install && mvn verify -pl gremlin-server -DskipIntegrationTests=false`. Also ran some manual tests and did some memory profiling of the driver to make sure there were no leaks. VOTE +1 You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/tinkerpop TINKERPOP-1249 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tinkerpop/pull/433.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #433 commit d881484a40ef7c5924e97a1adce7d0a7bf6654ea Author: Stephen Mallette <spmva@genoprime.com> Date: 2016-09-22T11:48:59Z Add keep-alive functionality to Java Driver.
          Hide
          k4rthikr@gmail.com Karthik Rajan added a comment -

          Thanks Stephen.

          Show
          k4rthikr@gmail.com Karthik Rajan added a comment - Thanks Stephen.
          Hide
          spmallette stephen mallette added a comment -

          Then doing g.V().limit(1) is fine for that purpose i suppose. I'd still say cheaper is better, so I'd maybe just go with: g.V().hasNext() which will just return true instead of forcing serialization of a vertex.

          > Which bring me to my followup question, is it possible to have an access restricted graph, which I can use for such things and not expose it out to clients?

          I don't think so - at least not directly with Gremlin Server.

          Show
          spmallette stephen mallette added a comment - Then doing g.V().limit(1) is fine for that purpose i suppose. I'd still say cheaper is better, so I'd maybe just go with: g.V().hasNext() which will just return true instead of forcing serialization of a vertex. > Which bring me to my followup question, is it possible to have an access restricted graph, which I can use for such things and not expose it out to clients? I don't think so - at least not directly with Gremlin Server.
          Hide
          k4rthikr@gmail.com Karthik Rajan added a comment -

          Fair point. Regarding the ping, I would like the ping to make sure that my implementation of Graph works. In the mysql world, an equal feature would be to have an access controlled admin database (ADMINDB), and ping is a simple INSERT into ADMINDB. This tests the engine and the storage in a simple operation. Which bring me to my followup question, is it possible to have an access restricted graph, which I can use for such things and not expose it out to clients?

          Show
          k4rthikr@gmail.com Karthik Rajan added a comment - Fair point. Regarding the ping, I would like the ping to make sure that my implementation of Graph works. In the mysql world, an equal feature would be to have an access controlled admin database (ADMINDB), and ping is a simple INSERT into ADMINDB. This tests the engine and the storage in a simple operation. Which bring me to my followup question, is it possible to have an access restricted graph, which I can use for such things and not expose it out to clients?
          Hide
          spmallette stephen mallette added a comment -

          I don't think we need a special ping API in Gremlin Server, if using websockets. websockets already has one, so i would think we would just want to utilize that. In the meantime i would manually send a request over the driver. I think that g.V().limit(1) is more expensive a ping than you need. I would just send a "null" or "0" or something like that.

          Show
          spmallette stephen mallette added a comment - I don't think we need a special ping API in Gremlin Server, if using websockets. websockets already has one, so i would think we would just want to utilize that. In the meantime i would manually send a request over the driver. I think that g.V().limit(1) is more expensive a ping than you need. I would just send a "null" or "0" or something like that.
          Hide
          k4rthikr@gmail.com Karthik Rajan added a comment -

          Before we get the driver to fire periodic pings, is it possible to make the GremlinServer open a ping API that can be wired with the load balancer? I'm working on monitoring the server health from within the host, and the current plan is to use a Driver to manually do something like g.V().limit(1) to see if the server is healthy. Do you recommend any other workarounds in the meanwhile?

          Show
          k4rthikr@gmail.com Karthik Rajan added a comment - Before we get the driver to fire periodic pings, is it possible to make the GremlinServer open a ping API that can be wired with the load balancer? I'm working on monitoring the server health from within the host, and the current plan is to use a Driver to manually do something like g.V().limit(1) to see if the server is healthy. Do you recommend any other workarounds in the meanwhile?

            People

            • Assignee:
              spmallette stephen mallette
              Reporter:
              phani1kumar Venkata Phani Kumar Mangipudi
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development