Got the following stacktrace for a large QueryPlan using WholeStageCodeGen:
After disabling codegen, everything works.
The root cause seems to be that we are trying to call the protected append method of BufferedRowIterator from an inner-class of a sub-class that is loaded by a different class-loader (after codegen compilation).
https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-5.html#jvms-5.4.4 states that a protected method R can be accessed only if one of the following two conditions is fulfilled:
- R is protected and is declared in a class C, and D is either a subclass of C or C itself. Furthermore, if R is not static, then the symbolic reference to R must contain a symbolic reference to a class T, such that T is either a subclass of D, a superclass of D, or D itself.
- R is either protected or has default access (that is, neither public nor protected nor private), and is declared by a class in the same run-time package as D.
2.) doesn't apply as we have loaded the class with a different class loader (and are in a different package) and 1.) doesn't apply because we are apparently trying to call the method from an inner class of a subclass of BufferedRowIterator.
Looking at the Code path of WholeStageCodeGen, the following happens:
- In WholeStageCodeGen, we create the subclass of BufferedRowIterator, along with a processNext method for processing the output of the child plan.
- In the child, which is a HashAggregateExec, we create the method which shows up at the top of the stack trace (called doAggregateWithKeysOutput )
- We add this method to the compiled code invoking addNewFunction of CodeGenerator
In the generated function body we call the append method.|
Now, the addNewFunction method states that:
This indeed seems to happen: the doAggregateWithKeysOutput method is put into a new private inner class. Thus, it doesn't have access to the protected append method anymore but still tries to call it, which results in the IllegalAccessError.
- Pass in the inlineToOuterClass flag when invoking the addNewFunction
- Make the append method public
- Re-declare the append method in the generated subclass (just invoking super). This way, inner classes should have access to it.