Setting -Djava.rmi.server.hostname=localhost does not work around this problem.
JMXManagedObjectRegistry manually builds an external JMX service URL using the value returned from InetAddress.getLocalHost().getHostName(). On the (Jenkins Solaris) box with the illegal hostname, the resulting URL is illegal and fails its own validation (method (JMXServiceURL#validate()). java.rmi.server.hostname is not considered.
It seems to me that it would be a useful enhancement if JMXManagedObjectRegistry used the contents of java.rmi.server.hostname when building the external URL and only fell back to getHostName() when unavailable. But the internal url should continue to use InetAddress.getLocalHost().getHostName() (so would still fail with illegal hostnames).
I think we should pursue getting the Solaris hostname corrected. Is an email to the APache infrastructure list best?