[FLINK-22788] Code of equals method grows beyond 64 KB

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.12.2
• Fix Version/s: 1.14.0, 1.12.5, 1.13.3
• Component/s: Table SQL / Runtime
• Labels: pull-request-available
• Assignee: Ingo Bürk
• Reporter: Maciej Bryński
• Created: 27/May/21 09:57
• Updated: 14/Apr/22 07:30
• Resolved: 22/Jun/21 17:18

Description

I'm getting following exception when running SQL with many (more than 500) columns

java.lang.RuntimeException: Could not instantiate generated class 'GroupAggValueEqualiser$38011'
        at org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:56) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.table.runtime.operators.aggregate.GroupAggFunction.open(GroupAggFunction.java:112) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:34) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]        at org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.streaming.api.operators.KeyedProcessOperator.open(KeyedProcessOperator.java:55) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:428) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:543) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:533) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:573) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.runtime.taskmanager.Task.run(Task.java:570) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_282]
Caused by: org.apache.flink.util.FlinkRuntimeException: org.apache.flink.api.common.InvalidProgramException: Table program cannot be compiled. This is a bug. Please file an issue.
        at org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:76) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:77) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:50) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        ... 12 more
Caused by: org.apache.flink.shaded.guava18.com.google.common.util.concurrent.UncheckedExecutionException: org.apache.flink.api.common.InvalidProgramException: Table program cannot be compiled. This is a bug. Please file an issue.
        at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]        at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:74) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:77) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:50) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        ... 12 more
Caused by: org.apache.flink.api.common.InvalidProgramException: Table program cannot be compiled. This is a bug. Please file an issue.
        at org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:89) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]        at org.apache.flink.table.runtime.generated.CompileUtils.lambda$compile$1(CompileUtils.java:74) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]        at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:74) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:77) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:50) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        ... 12 more
Caused by: org.codehaus.janino.InternalCompilerException: Compiling "GroupAggValueEqualiser$38011": Code of method "equals(Lorg/apache/flink/table/data/RowData;Lorg/apache/flink/table/data/RowData;)Z" of class "GroupAggValueEqualiser$38011" grows beyond 64 KB
        at org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:382) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:237) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:465) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:216) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:207) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:80) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.codehaus.commons.compiler.Cookable.cook(Cookable.java:75) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:86) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]        at org.apache.flink.table.runtime.generated.CompileUtils.lambda$compile$1(CompileUtils.java:74) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4742) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache.get(LocalCache.java:3937) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]        at org.apache.flink.shaded.guava18.com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4739) ~[flink-dist_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:74) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:77) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        at org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:50) ~[flink-table-blink_2.12-1.12.2-stream2.jar:1.12.2-stream2]
        ... 12 more

Issue Links

• is related to: FLINK-24480 EqualiserCodeGeneratorTest fails on azure
• relates to: FLINK-23007 Introduce java code splitter to split generated code
• links to: GitHub Pull Request #16213