Uploaded image for project: 'TinkerPop'
  1. TinkerPop
  2. TINKERPOP-2765

Race condition during script creation when using UnifiedChannelizer

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 3.6.0, 3.5.3, 3.6.1, 3.5.4
    • 3.7.0, 3.6.2, 3.5.5
    • server
    • None

    Description

      Running the following traversals in parallel in Gremlin Go with the UnifiedChannelizer in 3.5.x and 3.6.x Gremlin servers leads to a floating error, likely a race condition during script generation:

      g.With("evaluationTimeout", 10).Inject(1).SideEffect(&Lambda{"Thread.sleep(5000)", "gremlin-groovy"}).Iterate()
      g.With("evaluationTimeout", 10000).Inject(1).SideEffect(&Lambda{"Thread.sleep(5000)", "gremlin-groovy"}).Iterate()
      g.With("evaluationTimeout", 10000).With("evaluationTimeout", 10).Inject(1).SideEffect(&Lambda{"Thread.sleep(5000)", "gremlin-groovy"}).Iterate()
      g.WithStrategies(OptionsStrategy(map[string]interface{}{"evaluationTimeout": 10})).Inject(1).SideEffect(&Lambda{"Thread.sleep(5000)", "gremlin-groovy"}).Itearte()
      g.WithStrategies(OptionsStrategy(map[string]interface{}{"evaluationTimeout": 10000})).Inject(1).SideEffect(&Lambda{"Thread.sleep(5000)", "gremlin-groovy"}).Iterate() 

      Error traces:

      Here we see a hybrid traversal generated that leads to compilation error.  

      gremlin-test-server             | [ERROR] GremlinGroovyScriptEngine$GroovyCacheLoader - Script compilation FAILED gremlinscriptengine__ggremlinscriptengine__g.with.with("evaluationTimeout","evaluationTimeout",10000L10L))..injectinject((1L1L)).sideEffect(.sideEffect({Thread.sleep(5000)}).none(){Thread.sleep(5000)}).none() took 3ms {}
      gremlin-test-server             | org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
      gremlin-test-server             | Script4.groovy: 1: expecting ')', found '10L' @ line 1, column 102.
      gremlin-test-server             |    ut","evaluationTimeout",10000L10L))..inj
      gremlin-test-server             |                                  ^
      gremlin-test-server             |
      gremlin-test-server             | 1 error
      gremlin-test-server             |
      gremlin-test-server             |       at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:311)
      gremlin-test-server             |       at org.codehaus.groovy.control.ErrorCollector.addFatalError(ErrorCollector.java:151)
      gremlin-test-server             |       at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:121)
      gremlin-test-server             |       at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:133) 
      org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
      Script4.groovy: 1: unexpected token: . @ line 1, column 53.
         .inject((int) 1).sideE.inject(.inject((i
                                       ^1 error        at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:311)
              at org.codehaus.groovy.control.ErrorCollector.addFatalError(ErrorCollector.java:151)
              at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:121)
              at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:133)
              at org.codehaus.groovy.control.SourceUnit.addError(SourceUnit.java:325)
              at org.codehaus.groovy.antlr.AntlrParserPlugin.transformCSTIntoAST(AntlrParserPlugin.java:224)
              at org.codehaus.groovy.antlr.AntlrParserPlugin.parseCST(AntlrParserPlugin.java:192)
              at org.codehaus.groovy.control.SourceUnit.parse(SourceUnit.java:226)
              at org.codehaus.groovy.control.CompilationUnit$1.call(CompilationUnit.java:201)
              at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:965)
              at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:642)
              at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:618)
              at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:595)
              at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:401)
              at groovy.lang.GroovyClassLoader.access$300(GroovyClassLoader.java:89)
              at groovy.lang.GroovyClassLoader$5.provide(GroovyClassLoader.java:341)
              at groovy.lang.GroovyClassLoader$5.provide(GroovyClassLoader.java:338)
              at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:147)
              at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:336)
              at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:320)
              at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:262)
              at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine$GroovyCacheLoader.lambda$load$0(GremlinGroovyScriptEngine.java:821)
              at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
              at java.base/java.util.concurrent.CompletableFuture.asyncSupplyStage(CompletableFuture.java:1714)
              at java.base/java.util.concurrent.CompletableFuture.supplyAsync(CompletableFuture.java:1931)
              at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine$GroovyCacheLoader.load(GremlinGroovyScriptEngine.java:819)
              at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine$GroovyCacheLoader.load(GremlinGroovyScriptEngine.java:814)
              at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:3117)
              at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:144)
              at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$16(BoundedLocalCache.java:1968)
              at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1908)
              at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1966)
              at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1949)
              at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:113)
              at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:67)
              at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.getScriptClass(GremlinGroovyScriptEngine.java:569)
              at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:376)
              at java.scripting/javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
              at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:298)
              at org.apache.tinkerpop.gremlin.server.handler.AbstractSession.fromBytecode(AbstractSession.java:443)
              at org.apache.tinkerpop.gremlin.server.handler.AbstractSession.process(AbstractSession.java:264)
              at org.apache.tinkerpop.gremlin.server.handler.SingleTaskSession.run(SingleTaskSession.java:70)
              at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
              at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
              at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
              at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
              at java.base/java.lang.Thread.run(Thread.java:829) 

       

      Also reproducible in Python (https://github.com/apache/tinkerpop/pull/1700#discussion_r903067178):

      repro.py

      #!/usr/bin/env python3
      
      from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection
      from gremlin_python.process.anonymous_traversal import traversal
      
      g = traversal().with_remote(DriverRemoteConnection('ws://localhost:45940/gremlin','g'))
      
      try:
          g.inject(1).sideEffect(lambda: "Thread.sleep(1000)").iterate()
      except Exception as e:
          print(e) 

      stress.bash

      #!/bin/bash for i in $(seq 0 250); do python3 repro.py & pids[${i}]=$! done wait ${pids[*]} 

      Result:

      Similar hybrid traversal generated that leads to compilation error.  

      $ ./stress.bash
      
      Received error message '{'requestId': '0a358421-df3c-4965-8535-bf66337289c8', 'status': {'message': 'startup failed:\nScript4.groovy: 1: unexpected token: 1 @ line 1, column 44.\n1 error\n', 'code': 597, 'attributes': {'stackTrace': 'org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:\nScript4.groovy: 1: unexpected token: 1 @ line 1, column 44.\n   engine__g.inject((int) 1(int) 1).sideEff\n                                 ^\n\n1 error\n\n\tat org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:311)\n\tat org.codehaus.groovy.control.ErrorCollector.addFatalError(ErrorCollector.java:151)\n\tat org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:121)\n\tat org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:133)\n\tat org.codehaus.groovy.control.So
      ... 

      server log:

      [ERROR] ? - Script compilation FAILED gremlinscriptengine__g.inject((int) 1(int) 1).sideEffect().sideEffect({Thread.sleep(1000)}{Thread.sleep(1000)}))..nonenone()() took 11ms {}
      org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
      Script4.groovy: 1: unexpected token: 1 @ line 1, column 44.
         engine__g.inject((int) 1(int) 1).sideEff
                                       ^
      
      1 error
      
              at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:311)
              at org.codehaus.groovy.control.ErrorCollector.addFatalError(ErrorCollector.java:151)
              at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:121)
              at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:133)
              at org.codehaus.groovy.control.SourceUnit.addError(SourceUnit.java:325)
      ... 

       

      First reported during PR discussion: https://github.com/apache/tinkerpop/pull/1700#discussion_r902485983

      The race condition does not happen when using the default `WebSocketChannelizer` or the `WsAndHttpChannelizer`. 

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            spmallette Stephen Mallette
            xiazcy Yang Xia
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment