The kafka broker is just an object that takes a java.util.Properties in its constructor so it should be very easy to embed. Was there a reason it couldn't be embedded? We do this in tests and indeed this is how linkedin does it so i think it does work...
I think this is an important use case, the question is do we want to try to build this in or let other people create their own wrappers. I could see going either way, but my experience is that a lot of this packaging stuff tends to be fairly environment specific so even if someone else is using mx4j they may have a different incompatible version so it is just better to let people embed and do their own thing. (Actually LinkedIn is using mx4j in this very way and I am a little concerned this would conflict with out usage...so it would be good if there was an option so it could be disabled at the least).
So I think we should either:
1. Document how to embed the broker to make it easy for people to do these kinds of things OR
2. Make it so that this can be disabled