Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-3213

Java client should attempt a different tablet server when retrying during tserver quiescing

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.15.0
    • Component/s: java
    • Labels:
      None

      Description

      One of our clusters ran into the following error message when leaving a tablet server quiesced for an extended period of time:

      ERROR Runner: Pipeline exception occurred: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 6.0 failed 4 times, most recent failure: Lost task 1.3 in stage 6.0 (TID 1922, tserver-018.edh.company.com, executor 58): org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: ScanRequest(scannerId=null, tablet=9e17b554f85f4a7f855771d8b5c913f5, attempt=24, KuduRpc(method=Scan, tablet=9e17b554f85f4a7f855771d8b5c913f5, attempt=24, DeadlineTracker(timeout=30000, elapsed=27988), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC GetTableLocations: sending RPC to server master-name-003.edh.company.com:7051, [12ms] Sub RPC GetTableLocations: received response from server master-name-003.edh.company.com:7051: OK, [22ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [116ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [117ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [126ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [129ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [129ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [146ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [149ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [149ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [166ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [168ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [168ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [206ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [209ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [209ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [266ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [268ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [268ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [306ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [308ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [308ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [545ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [548ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [548ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [865ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [868ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [868ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [1266ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [1269ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [1269ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [2626ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [2628ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [2628ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [4746ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [4749ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [4749ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [8206ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [8209ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [8209ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [8626ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [8629ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [8629ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [11746ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [11750ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [11750ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [14006ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [14008ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [14008ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [16366ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [16369ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [16369ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [19786ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [19788ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [19788ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [20106ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [20108ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [20108ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [22366ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [22369ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [22369ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [23946ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [23948ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [23949ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [27266ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [27269ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [27269ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [27986ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [27988ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0)))ERROR Runner: Pipeline exception occurred: org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 6.0 failed 4 times, most recent failure: Lost task 1.3 in stage 6.0 (TID 1922, tserver-018.edh.company.com, executor 58): org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: ScanRequest(scannerId=null, tablet=9e17b554f85f4a7f855771d8b5c913f5, attempt=24, KuduRpc(method=Scan, tablet=9e17b554f85f4a7f855771d8b5c913f5, attempt=24, DeadlineTracker(timeout=30000, elapsed=27988), Traces: [0ms] refreshing cache from master, [1ms] Sub RPC GetTableLocations: sending RPC to server master-name-003.edh.company.com:7051, [12ms] Sub RPC GetTableLocations: received response from server master-name-003.edh.company.com:7051: OK, [22ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [116ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [117ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [126ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [129ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [129ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [146ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [149ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [149ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [166ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [168ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [168ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [206ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [209ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [209ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [266ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [268ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [268ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [306ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [308ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [308ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [545ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [548ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [548ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [865ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [868ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [868ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [1266ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [1269ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [1269ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [2626ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [2628ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [2628ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [4746ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [4749ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [4749ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [8206ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [8209ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [8209ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [8626ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [8629ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [8629ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [11746ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [11750ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [11750ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [14006ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [14008ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [14008ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [16366ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [16369ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [16369ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [19786ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [19788ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [19788ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [20106ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [20108ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [20108ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [22366ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [22369ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [22369ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [23946ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [23948ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [23949ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [27266ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [27269ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0), [27269ms] received response from server e1a4405443d845249b5ed15c8e882211: Service unavailable: Tablet server is quiescing (error 0), [27986ms] sending RPC to server e1a4405443d845249b5ed15c8e882211, [27988ms] delaying RPC due to: Service unavailable: Tablet server is quiescing (error 0))) at org.apache.kudu.client.KuduException.transformException(KuduException.java:110) at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:402) at org.apache.kudu.client.KuduScanner.nextRows(KuduScanner.java:57) at org.apache.kudu.spark.kudu.RowIterator.hasNext(KuduRDD.scala:153) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409) at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage11.processNext(Unknown Source) at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43) at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$13$$anon$1.hasNext(WholeStageCodegenExec.scala:636) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:409)

      It seems like the handling of the TABLET_NOT_RUNNING code is different between the Java and C++ clients. In the C++ client, the tserver is blacklisted and another tserver is attempted. In the Java client, the request appears to simply be retried at the same server, as it would a SERVICE_UNAVAILABLE error.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              awong Andrew Wong
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: