Right now a communication exception is thrown by the server when a malformed DN is given in a PDU. I tested this with bind for example using a bad BindDN. When the ASN1 codec fails the server cannot really tell the difference between a bad protocol PDU which causes a protocolError (2) resulting in a JNDI CommunicationException from a bad DN which should return a resultCode of namingViolation (64) which in JNDI comes back to the client as an InvalidNameException. For more info on resultCodes mapping to JNDI exceptions see:
http://java.sun.com/products/jndi/jndi-ldap-gl.html#EXCEPT
Basically the ASN1 decoder has to throw exceptions with hints regarding the failure to allow the server to respond appropriately to the client. This can be done by embedding additional info such as a result code in an subclass of DecoderException. Then the LdapProtocolProvider can access this info.
This problem is a direct result of trying to parse a DN for correctness when this is not the responsibility of the ASN1 codec. The LDAP BER codec should be giving stuff back to the server as is and letting the server determine whether or not the dn or other (non-asn1 constrained) constructs are invalid.