When a closure sets the private field of a class, it does it via a ScriptBytecodeAdapter.setGroovyObjectProperty on the closure object itself, whereas when getting a private field of the class it uses a synthetic "pfaccess$0" method.
Decompiled result of closure class (via Fernflower in IntelliJ):
Workaround is to remove "private" keyword on the variable declaration, which causes Groovy to generate public getter and setter. In that mode, the static compiler does generate direct calls to both the getter and the setter. In one example within my project the performance speedup was 10x with this workaround. The only thing I notice in the workaround that is odd is that instead of normal Java casts used, ScriptBytecodeAdapter.castToType is used on "getThisObject" before calling the getters or setters.
This issue could possibly related to GROOVY-6825, but in that case it's an inner class, and the issue description (currently) is not clear enough to see if it's also generating a setProperty call.