Major Description
One of our clusters ran into the following error message when leaving a tablet server quiesced for an extended period of time:
ERROR Runner: Pipeline exception occurred: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 6.0 failed 4 times, most recent failure: Lost task 1.3 in stage 6.0 (TID 1922, tserver-018.edh.company.com, executor 58): org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: ScanRequest(scannerId=null, tablet=9e17b554f85f4a7f855771d8b5c913f5, attempt=24, KuduRpc(method=Scan, tablet=9e17b554f85f4a7f855771d8b5c913f5, attempt=24, DeadlineTracker(timeout=30000, elapsed=27988), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC GetTableLocations: sending RPC to server master-name-003.edh.company.com:7051, [12ms] Sub RPC GetTableLocations: received response from server master-name-003.edh.company.com:7051: OK, [22ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [116ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [117ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [126ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [129ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [129ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [146ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [149ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [149ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [166ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [168ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [168ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [206ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [209ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [209ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [266ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [268ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [268ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [306ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [308ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [308ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [545ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [548ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [548ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [865ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [868ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [868ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [1266ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [1269ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [1269ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [2626ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [2628ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [2628ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [4746ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [4749ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [4749ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [8206ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [8209ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [8209ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [8626ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [8629ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [8629ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [11746ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [11750ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [11750ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [14006ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [14008ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [14008ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [16366ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [16369ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [16369ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [19786ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [19788ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [19788ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [20106ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [20108ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [20108ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [22366ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [22369ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [22369ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [23946ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [23948ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [23949ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [27266ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [27269ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [27269ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [27986ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [27988ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0)))ERROR Runner: Pipeline exception occurred: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 6.0 failed 4 times, most recent failure: Lost task 1.3 in stage 6.0 (TID 1922, tserver-018.edh.company.com, executor 58): org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: ScanRequest(scannerId=null, tablet=9e17b554f85f4a7f855771d8b5c913f5, attempt=24, KuduRpc(method=Scan, tablet=9e17b554f85f4a7f855771d8b5c913f5, attempt=24, DeadlineTracker(timeout=30000, elapsed=27988), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC GetTableLocations: sending RPC to server master-name-003.edh.company.com:7051, [12ms] Sub RPC GetTableLocations: received response from server master-name-003.edh.company.com:7051: OK, [22ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [116ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [117ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [126ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [129ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [129ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [146ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [149ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [149ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [166ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [168ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [168ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [206ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [209ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [209ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [266ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [268ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [268ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [306ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [308ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [308ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [545ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [548ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [548ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [865ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [868ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [868ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [1266ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [1269ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [1269ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [2626ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [2628ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [2628ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [4746ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [4749ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [4749ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [8206ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [8209ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [8209ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [8626ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [8629ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [8629ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [11746ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [11750ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [11750ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [14006ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [14008ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [14008ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [16366ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [16369ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [16369ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [19786ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [19788ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [19788ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [20106ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [20108ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [20108ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [22366ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [22369ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [22369ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [23946ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [23948ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [23949ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [27266ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [27269ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [27269ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [27986ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [27988ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0))) at org.apache.kudu.client.KuduException.transformException(KuduException.java:110) at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:402) at org.apache.kudu.client.KuduScanner.nextRows(KuduScanner.java:57) at org.apache.kudu.spark.kudu.RowIterator.hasNext(KuduRDD.scala:153) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage11.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)
It seems like the handling of the TABLET_NOT_RUNNING code is different between the Java and C++ clients. In the C++ client, the tserver is blacklisted and another tserver is attempted. In the Java client, the request appears to simply be retried at the same server, as it would a SERVICE_UNAVAILABLE error.