JAASRealm cannot be used with DigestAuthenticator. JAASRealm only overrides authenticate(String, String), whereas DigestAuthenticator calls authenticate(String, String, String, String, String, String, String, String). This is similar to bug 41407, where CLIENT-CERT authentication is used.
This has been fixed in trunk and proposed for 5.5.x and 6.0.x
This has been fixed in 6.0.x and will be included in 6.0.19 onwards.
This has been fixed in 5.5.x and will be included in 5.5.28 onwards.