[FLINK-16589] Flink Table SQL fails/crashes with big queries with lots of fields

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.10.0
• Fix Version/s: 1.11.0
• Component/s: Table SQL / Planner
• Labels: pull-request-available
• Assignee: Benchao Li
• Reporter: Viet Pham
• Created: 13/Mar/20 11:21
• Updated: 16/Jun/21 07:28
• Resolved: 19/Jun/20 08:20

Description

Hi,

My use case is a streaming application with a few streaming tables.

I was trying to build a SELECT query (and registering it as a temporary view) with about 200 fields/expressions out of another streaming table. The application is successfully submitted to Flink cluster. However the worker processes keep crashing, with the exception as quoted below. 

It clearly mentioned in the log that this is a bug, so I fire this ticket. By the way, if I lower the number of fields down to 100 then it works nicely.

Please advice.

Thanks a lot for all the efforts bring Flink up. It is really amazing!

java.lang.RuntimeException: Could not instantiate generated class 'GroupAggsHandler$9687'    at org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:57)    at org.apache.flink.table.runtime.operators.aggregate.MiniBatchGroupAggFunction.open(MiniBatchGroupAggFunction.java:136)    at org.apache.flink.table.runtime.operators.bundle.AbstractMapBundleOperator.open(AbstractMapBundleOperator.java:84)    at org.apache.flink.streaming.runtime.tasks.StreamTask.initializeStateAndOpen(StreamTask.java:1007)    at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:454)    at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)    at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:449)    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:461)    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)    at java.lang.Thread.run(Thread.java:748)Caused by: org.apache.flink.util.FlinkRuntimeException: org.apache.flink.api.common.InvalidProgramException: Table program cannot be compiled. This is a bug. Please file an issue.    at org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:68)    at org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:78)    at org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:52)    ... 10 moreCaused by: org.apache.flink.shaded.guava18.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.flink.api.common.InvalidProgramException: Table program cannot be compiled. This is a bug. Please file an issue.    at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203)    at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.get(LocalCache.java:3937)    at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739)    at org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:66)    ... 12 moreCaused by: org.apache.flink.api.common.InvalidProgramException: Table program cannot be compiled. This is a bug. Please file an issue.    at org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:81)    at org.apache.flink.table.runtime.generated.CompileUtils.lambda$compile$1(CompileUtils.java:66)    at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742)    at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)    at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)    at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)    at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)    ... 15 moreCaused by: org.codehaus.janino.InternalCompilerException: Compiling "GroupAggsHandler$9687": Code of method "retract(Lorg/apache/flink/table/dataformat/BaseRow;)V" of class "GroupAggsHandler$9687" grows beyond 64 KB    at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:382)    at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:237)    at org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:465)    at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:216)    at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:207)    at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:80)    at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:75)    at org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:78)    ... 21 moreCaused by: org.codehaus.janino.InternalCompilerException: Code of method "retract(Lorg/apache/flink/table/dataformat/BaseRow;)V" of class "GroupAggsHandler$9687" grows beyond 64 KB    at org.codehaus.janino.CodeContext.makeSpace(CodeContext.java:1009)    at org.codehaus.janino.CodeContext.write(CodeContext.java:901)    at org.codehaus.janino.CodeContext.writeShort(CodeContext.java:1026)    at org.codehaus.janino.UnitCompiler.writeConstantLongInfo(UnitCompiler.java:12274)    at org.codehaus.janino.UnitCompiler.pushConstant(UnitCompiler.java:10679)    at org.codehaus.janino.UnitCompiler.compileGet2(UnitCompiler.java:4936)    at org.codehaus.janino.UnitCompiler.access$8400(UnitCompiler.java:215)    at org.codehaus.janino.UnitCompiler$16.visitUnaryOperation(UnitCompiler.java:4414)    at org.codehaus.janino.UnitCompiler$16.visitUnaryOperation(UnitCompiler.java:4394)    at org.codehaus.janino.Java$UnaryOperation.accept(Java.java:4719)    at org.codehaus.janino.UnitCompiler.compileGet(UnitCompiler.java:4394)    at org.codehaus.janino.UnitCompiler.fakeCompile(UnitCompiler.java:3719)    at org.codehaus.janino.UnitCompiler.compileGetValue(UnitCompiler.java:5569)    at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:2580)    at org.codehaus.janino.UnitCompiler.access$2700(UnitCompiler.java:215)    at org.codehaus.janino.UnitCompiler$6.visitLocalVariableDeclarationStatement(UnitCompiler.java:1503)    at org.codehaus.janino.UnitCompiler$6.visitLocalVariableDeclarationStatement(UnitCompiler.java:1487)    at org.codehaus.janino.Java$LocalVariableDeclarationStatement.accept(Java.java:3511)    at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:1487)    at org.codehaus.janino.UnitCompiler.compileStatements(UnitCompiler.java:1567)    at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:3388)    at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1357)    at org.codehaus.janino.UnitCompiler.compileDeclaredMethods(UnitCompiler.java:1330)    at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:822)    at org.codehaus.janino.UnitCompiler.compile2(UnitCompiler.java:432)    at org.codehaus.janino.UnitCompiler.access$400(UnitCompiler.java:215)    at org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:411)    at org.codehaus.janino.UnitCompiler$2.visitPackageMemberClassDeclaration(UnitCompiler.java:406)    at org.codehaus.janino.Java$PackageMemberClassDeclaration.accept(Java.java:1414)    at org.codehaus.janino.UnitCompiler.compile(UnitCompiler.java:406)    at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:378)    ... 28 more

Issue Links

• relates to: FLINK-23007 Introduce java code splitter to split generated code
• links to: GitHub Pull Request #11512