If the STRING argument to ap_custom_response() begins with a double quote,
it prepends another double quote.  This second double quote is later stripped
by ap_send_error_response(), but the original one remains and is included in the
message. I suspect that users of ap_custom_response() expect it to treat its
STRING argument in the same way as the documentation describes the
ErrorDocument directive's second argument.

ap_custom_response() uses this code:
  conf->response_code_strings[idx] =
     ((ap_is_url(string) || (*string == '/')) && (*string != '"')) ?
       apr_pstrdup(r->pool, string) : apr_pstrcat(r->pool, "\"", string, NULL);
The test in the conditional expression doesn't make much sense.
If the LHS of the conjunction is true the RHS must also be true, and if the LHS
is false, the RHS is irrelevant.  If *string == '"' the test is false and
a double quote will be prepended.

I believe the expression should be:
  (ap_is_url(string) || *string == '/' || *string == '"') ?
This will add a double quote if the argument is not a URL and not a path and
not something that already begins with one.  The double quote will be stripped
by ap_send_error_response(), yielding the characters that follow as the message.
I suspect that this is the intended (and correct) behaviour.