Details
Major Description
Running the following traversals in parallel in Gremlin Go with the UnifiedChannelizer in 3.5.x and 3.6.x Gremlin servers leads to a floating error, likely a race condition during script generation:
g.With("evaluationTimeout", 10).Inject(1).SideEffect(&Lambda{"Thread.sleep(5000)", "gremlin-groovy"}).Iterate() g.With("evaluationTimeout", 10000).Inject(1).SideEffect(&Lambda{"Thread.sleep(5000)", "gremlin-groovy"}).Iterate() g.With("evaluationTimeout", 10000).With("evaluationTimeout", 10).Inject(1).SideEffect(&Lambda{"Thread.sleep(5000)", "gremlin-groovy"}).Iterate() g.WithStrategies(OptionsStrategy(map[string]interface{}{"evaluationTimeout": 10})).Inject(1).SideEffect(&Lambda{"Thread.sleep(5000)", "gremlin-groovy"}).Itearte() g.WithStrategies(OptionsStrategy(map[string]interface{}{"evaluationTimeout": 10000})).Inject(1).SideEffect(&Lambda{"Thread.sleep(5000)", "gremlin-groovy"}).Iterate()
Error traces:
Here we see a hybrid traversal generated that leads to compilation error.
gremlin-test-server | [ERROR] GremlinGroovyScriptEngine$GroovyCacheLoader - Script compilation FAILED gremlinscriptengine__ggremlinscriptengine__g.with.with("evaluationTimeout","evaluationTimeout",10000L10L))..injectinject((1L1L)).sideEffect(.sideEffect({Thread.sleep(5000)}).none(){Thread.sleep(5000)}).none() took 3ms {} gremlin-test-server | org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: gremlin-test-server | Script4.groovy: 1: expecting ')', found '10L' @ line 1, column 102. gremlin-test-server | ut","evaluationTimeout",10000L10L))..inj gremlin-test-server | ^ gremlin-test-server | gremlin-test-server | 1 error gremlin-test-server | gremlin-test-server | at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:311) gremlin-test-server | at org.codehaus.groovy.control.ErrorCollector.addFatalError(ErrorCollector.java:151) gremlin-test-server | at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:121) gremlin-test-server | at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:133)
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: Script4.groovy: 1: unexpected token: . @ line 1, column 53. .inject((int) 1).sideE.inject(.inject((i ^1 error at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:311) at org.codehaus.groovy.control.ErrorCollector.addFatalError(ErrorCollector.java:151) at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:121) at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:133) at org.codehaus.groovy.control.SourceUnit.addError(SourceUnit.java:325) at org.codehaus.groovy.antlr.AntlrParserPlugin.transformCSTIntoAST(AntlrParserPlugin.java:224) at org.codehaus.groovy.antlr.AntlrParserPlugin.parseCST(AntlrParserPlugin.java:192) at org.codehaus.groovy.control.SourceUnit.parse(SourceUnit.java:226) at org.codehaus.groovy.control.CompilationUnit$1.call(CompilationUnit.java:201) at org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:965) at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:642) at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:618) at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:595) at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:401) at groovy.lang.GroovyClassLoader.access$300(GroovyClassLoader.java:89) at groovy.lang.GroovyClassLoader$5.provide(GroovyClassLoader.java:341) at groovy.lang.GroovyClassLoader$5.provide(GroovyClassLoader.java:338) at org.codehaus.groovy.runtime.memoize.ConcurrentCommonCache.getAndPut(ConcurrentCommonCache.java:147) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:336) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:320) at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:262) at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine$GroovyCacheLoader.lambda$load$0(GremlinGroovyScriptEngine.java:821) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) at java.base/java.util.concurrent.CompletableFuture.asyncSupplyStage(CompletableFuture.java:1714) at java.base/java.util.concurrent.CompletableFuture.supplyAsync(CompletableFuture.java:1931) at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine$GroovyCacheLoader.load(GremlinGroovyScriptEngine.java:819) at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine$GroovyCacheLoader.load(GremlinGroovyScriptEngine.java:814) at com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:3117) at com.github.benmanes.caffeine.cache.LocalCache.lambda$statsAware$0(LocalCache.java:144) at com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$16(BoundedLocalCache.java:1968) at java.base/java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1908) at com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1966) at com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1949) at com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:113) at com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:67) at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.getScriptClass(GremlinGroovyScriptEngine.java:569) at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:376) at java.scripting/javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233) at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:298) at org.apache.tinkerpop.gremlin.server.handler.AbstractSession.fromBytecode(AbstractSession.java:443) at org.apache.tinkerpop.gremlin.server.handler.AbstractSession.process(AbstractSession.java:264) at org.apache.tinkerpop.gremlin.server.handler.SingleTaskSession.run(SingleTaskSession.java:70) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)
Also reproducible in Python (https://github.com/apache/tinkerpop/pull/1700#discussion_r903067178):
repro.py
#!/usr/bin/env python3 from gremlin_python.driver.driver_remote_connection import DriverRemoteConnection from gremlin_python.process.anonymous_traversal import traversal g = traversal().with_remote(DriverRemoteConnection('ws://localhost:45940/gremlin','g')) try: g.inject(1).sideEffect(lambda: "Thread.sleep(1000)").iterate() except Exception as e: print(e)
stress.bash
#!/bin/bash for i in $(seq 0 250); do python3 repro.py & pids[${i}]=$! done wait ${pids[*]}
Result:
Similar hybrid traversal generated that leads to compilation error.
$ ./stress.bash Received error message '{'requestId': '0a358421-df3c-4965-8535-bf66337289c8', 'status': {'message': 'startup failed:\nScript4.groovy: 1: unexpected token: 1 @ line 1, column 44.\n1 error\n', 'code': 597, 'attributes': {'stackTrace': 'org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:\nScript4.groovy: 1: unexpected token: 1 @ line 1, column 44.\n engine__g.inject((int) 1(int) 1).sideEff\n ^\n\n1 error\n\n\tat org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:311)\n\tat org.codehaus.groovy.control.ErrorCollector.addFatalError(ErrorCollector.java:151)\n\tat org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:121)\n\tat org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:133)\n\tat org.codehaus.groovy.control.So ...
server log:
[ERROR] ? - Script compilation FAILED gremlinscriptengine__g.inject((int) 1(int) 1).sideEffect().sideEffect({Thread.sleep(1000)}{Thread.sleep(1000)}))..nonenone()() took 11ms {} org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed: Script4.groovy: 1: unexpected token: 1 @ line 1, column 44. engine__g.inject((int) 1(int) 1).sideEff ^ 1 error at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:311) at org.codehaus.groovy.control.ErrorCollector.addFatalError(ErrorCollector.java:151) at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:121) at org.codehaus.groovy.control.ErrorCollector.addError(ErrorCollector.java:133) at org.codehaus.groovy.control.SourceUnit.addError(SourceUnit.java:325) ...
First reported during PR discussion: https://github.com/apache/tinkerpop/pull/1700#discussion_r902485983
The race condition does not happen when using the default `WebSocketChannelizer` or the `WsAndHttpChannelizer`.