[SOLR-9492] Request status API returns a completed status even if the collection API call failed

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 5.5.2, 6.2
• Component/s: SolrCloud
• Labels: difficulty-medium impact-high
• Assignee: Unassigned
• Reporter: Shalin Shekhar Mangar
• Created: 09/Sep/16 16:05
• Updated: 22/Jul/20 20:01

Description

A failed split shard response is:

{success={127.0.0.1:43245_hfnp%2Fbq={responseHeader={status=0,QTime=2}},127.0.0.1:43245_hfnp%2Fbq={responseHeader={status=0,QTime=0}},127.0.0.1:43245_hfnp%2Fbq={responseHeader={status=0,QTime=0}},127.0.0.1:43245_hfnp%2Fbq={responseHeader={status=0,QTime=0}},127.0.0.1:43245_hfnp%2Fbq={responseHeader={status=0,QTime=0}},127.0.0.1:43245_hfnp%2Fbq={responseHeader={status=0,QTime=0}},127.0.0.1:43245_hfnp%2Fbq={responseHeader={status=0,QTime=0}},127.0.0.1:50948_hfnp%2Fbq={responseHeader={status=0,QTime=0}}},c32001ed-3bca-4ae0-baae-25a3c99e35e65883644576126044={responseHeader={status=0,QTime=0},STATUS=completed,Response=TaskId: c32001ed-3bca-4ae0-baae-25a3c99e35e65883644576126044 webapp=null path=/admin/cores params={async=c32001ed-3bca-4ae0-baae-25a3c99e35e65883644576126044&qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_0_replica1&action=CREATE&collection=collection1&shard=shard1_0&wt=javabin&version=2} status=0 QTime=2},c32001ed-3bca-4ae0-baae-25a3c99e35e65883647597130004={responseHeader={status=0,QTime=0},STATUS=completed,Response=TaskId: c32001ed-3bca-4ae0-baae-25a3c99e35e65883647597130004 webapp=null path=/admin/cores params={async=c32001ed-3bca-4ae0-baae-25a3c99e35e65883647597130004&qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_1_replica1&action=CREATE&collection=collection1&shard=shard1_1&wt=javabin&version=2} status=0 QTime=0},c32001ed-3bca-4ae0-baae-25a3c99e35e65883649607943904={responseHeader={status=0,QTime=0},STATUS=completed,Response=TaskId: c32001ed-3bca-4ae0-baae-25a3c99e35e65883649607943904 webapp=null path=/admin/cores params={nodeName=127.0.0.1:43245_hfnp%252Fbq&core=collection1_shard1_1_replica1&async=c32001ed-3bca-4ae0-baae-25a3c99e35e65883649607943904&qt=/admin/cores&coreNodeName=core_node6&action=PREPRECOVERY&checkLive=true&state=active&onlyIfLeader=true&wt=javabin&version=2} status=0 QTime=0},c32001ed-3bca-4ae0-baae-25a3c99e35e65883649612565003={responseHeader={status=0,QTime=0},STATUS=completed,Response=TaskId: c32001ed-3bca-4ae0-baae-25a3c99e35e65883649612565003 webapp=null path=/admin/cores params={core=collection1&async=c32001ed-3bca-4ae0-baae-25a3c99e35e65883649612565003&qt=/admin/cores&action=SPLIT&targetCore=collection1_shard1_0_replica1&targetCore=collection1_shard1_1_replica1&wt=javabin&version=2} status=0 QTime=0},c32001ed-3bca-4ae0-baae-25a3c99e35e65883650618358632={responseHeader={status=0,QTime=0},STATUS=completed,Response=TaskId: c32001ed-3bca-4ae0-baae-25a3c99e35e65883650618358632 webapp=null path=/admin/cores params={async=c32001ed-3bca-4ae0-baae-25a3c99e35e65883650618358632&qt=/admin/cores&name=collection1_shard1_1_replica1&action=REQUESTAPPLYUPDATES&wt=javabin&version=2} status=0 QTime=0},c32001ed-3bca-4ae0-baae-25a3c99e35e65883650636428900={responseHeader={status=0,QTime=0},STATUS=completed,Response=TaskId: c32001ed-3bca-4ae0-baae-25a3c99e35e65883650636428900 webapp=null path=/admin/cores params={async=c32001ed-3bca-4ae0-baae-25a3c99e35e65883650636428900&qt=/admin/cores&collection.configName=conf1&name=collection1_shard1_0_replica0&action=CREATE&collection=collection1&shard=shard1_0&wt=javabin&version=2} status=0 QTime=0},failure={127.0.0.1:43245_hfnp%2Fbq=org.apache.solr.client.solrj.SolrServerException:IOException occured when talking to server at: http://127.0.0.1:43245/hfnp/bq},Operation splitshard caused exception:=org.apache.solr.common.SolrException: ADDREPLICA failed to create replica,exception={msg=ADDREPLICA failed to create replica,rspCode=500}}

Note the "failure" bit. The split shard couldn't add a replica. But when you use the request status API, it returns a "completed" status.

Apparently, completed doesn't mean it was successful! In any case, it is very misleading and makes it very hard to properly use the Collection APIs. We need more investigation to figure out what other Collection APIs might be affected.