This approach will work.
I see another possibility that is simpler, but has some serious drawbacks.
Instead of separating the source directories and classpaths at build time, file set includes/excludes could be used at jar creation time to create two jar files.
Benefit: simpler, requires no code changes:
Drawback: does not actually disentangle the code, which is more brittle and could lead to confusion depending on what changes are in the user lib jars.
That is a big drawback.
The above approach is more complicated but does enforce true separation.
In the long run, tackling the disentangling of the lib from the system is a requirement but the extra complexity requires more careful review. It also is the bulk of the work required to make the lib a truly separate library – for maven or OSGi.
I'm not an expert on the details of the API and code changes needed for clean separation as a consequence of this. The callback changes seem like an improvement. Duplication of wrappers and reflective access are usually signs that there is a better way, but that better way almost always requires an API break.
Overall, I like this approach because it is moving in the right direction for separating the core execution from more volatile or user-extensible library code.