[SOLR-9143] Solr basic authentication randomly throwing "Invalid Key" error

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Bug
• Affects Version/s: 5.5
• Component/s: security
• Assignee: Noble Paul
• Reporter: Shamik Bandopadhyay
• Created: 21/May/16 20:09
• Updated: 24/May/16 05:11
• Resolved: 24/May/16 03:58

Description

I'm facing a weird issue where Basic authentications are failing randomly. The error is originating as "Invalid key" from PKIAuthenticationPlugin.java followed by missing userPrincipal. Here's the stacktrace:

ERROR923629[qtp466002798-20] -
org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:125)

• Invalid key
  INFO923630[qtp466002798-20] -
  org.apache.solr.security.RuleBasedAuthorizationPlugin.checkPathPerm(RuleBasedAuthorizationPlugin.java:144)
• request has come without principal. failed permission
  org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@1a343033
  INFO923630[qtp466002798-20] -
  org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:429) -
  USER_REQUIRED auth header null context : userPrincipal: [null] type:
  [READ], collections: [knowledge,], Path: [/select] path : /select params
  :df=text&distrib=false&qt=/select&preferLocalShards=false&fl=id&fl=score&shards.purpose=4&start=0&fsv=true&shard.url=
  http://xx.xxx.x.222:8983/solr/knowledge/|http://xx.xxx.xxx.246:8983/solr/knowledge/&rows=3&version=2&q=:&NOW=1463512962899&isShard=true&wt=javabin

My security.json

{
"authentication": {
"blockUnknown": false,
"class": "solr.BasicAuthPlugin",
"credentials":

{ "solr": "IV0EHq1OnNrj6gvRCwvFwTrZ1+z1oBbnQdiVC3otuq0= Ndd7LKvVBAaZIF0QAVi1ekCfAJXr1GGfLtRUXhgrF8c=" }

},
"authorization": {
"class": "solr.RuleBasedAuthorizationPlugin",
"user-role":

{ "solr": "admin", "solradmin": "admin", "beehive": "dev", "readuser": "read" }

,
"permissions": [

{ "name": "security-edit", "role": "admin" }

,

{ "name": "browse", "collection": "knowledge", "path": "/browse", "role": [ "admin", "dev", "read" ] }

,

{ "name": "select", "collection": "knowledge", "path": "/select", "role": [ "admin", "dev", "read" ] }

,

{ "name": "admin-ui", "path": "/", "role": [ "admin", "dev" ] }

,

{ "name": "update", "role": [ "admin", "dev" ] }

,

{ "name": "collection-admin-edit", "role": [ "admin" ] }

,

{ "name": "schema-edit", "role": [ "admin" ] }

,

{ "name": "config-edit", "role": [ "admin" ] }

]
}
}

Sample Java client:

SolrClient client = new CloudSolrClient("zoohost1:2181,zoohost2:2181,zoohost3:2181");
((CloudSolrClient)client).setDefaultCollection(DEFAULT_COLLECTION);
ModifiableSolrParams param = getSearchSolrQuery();
SolrRequest<?> solrRequest = new QueryRequest(param);
solrRequest.setBasicAuthCredentials(USER, PASSWORD);
try{
for(int j=0;j<20;j++)

{ NamedList results = client.request(solrRequest); }

}catch(Exception ex){

}

private static ModifiableSolrParams getSearchSolrQuery() {
ModifiableSolrParams solrParams = new ModifiableSolrParams();
solrParams.set("q", ":");
solrParams.set("qt","/select");
solrParams.set("rows", "3");
return solrParams;
}

Sometimes, the error is being thrown at the very first call, otherwise in the middle of the iteration. It's consistent with my custom user or the default "solr/SolrRocks". I even cleaned up the zookeeper data, started the cluster from fresh, uploaded the security.json, but without any luck.

Incidentally, I'm also seeing similar exception if I try to start and stop a node in the cluster while indexing is in process. Here's the log:

ERROR 19543[qtp466002798-21] - org.apache.solr.security.PKIAuthenticationPlugin.doAuthenticate(PKIAuthenticationPlugin.java:125) - Invalid key
INFO 19543[qtp466002798-21] - org.apache.solr.security.RuleBasedAuthorizationPlugin.checkPathPerm(RuleBasedAuthorizationPlugin.java:144) - request has come without principal. failed permission org.apache.solr.security.RuleBasedAuthorizationPlugin$Permission@101fe889
INFO 19543[qtp466002798-21] - org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:429) - USER_REQUIRED auth header null context : userPrincipal: [null] type: [WRITE], collections: [knowledge,], Path: [/update] path : /update params :update.distrib=FROMLEADER&distrib.from=http://xx.xxx.xxx.246:8983/solr/knowledge/&wt=javabin&version=2

Based on the source code, it seems like the error is generated due to timeout issues. I bumped up SOLR_OPTS="$SOLR_OPTS -Dpkiauth.ttl=50000" to 50 sec, but didn't make any difference.

My cluster contains 2 shards with 1 replica each.

I'll appreciate if someone can take a look and provide me some pointers.