[KUDU-2390] ITClient fails with "Row count unexpectedly decreased"

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: 1.7.0, 1.8.0
• Fix Version/s: NA
• Component/s: java, test
• Assignee: Unassigned
• Reporter: Todd Lipcon
• Created: 28/Mar/18 18:45
• Updated: 03/Jun/20 18:12
• Resolved: 03/Jun/20 18:12

Description

On master, hit the following failure of ITClient:

20:05:05.407 [DEBUG - New I/O worker #17] (AsyncKuduScanner.java:934) AsyncKuduScanner$Response(scannerId = "6ddf5d0da48241aea4b9eb51645716cc", data = RowResultIterator for 27600 rows, more = true, responseScanTimestamp = 6234957022375723008) for scanner
20:05:05.407 [DEBUG - New I/O worker #17] (AsyncKuduScanner.java:447) Scanner "6ddf5d0da48241aea4b9eb51645716cc" opened on d78cb5506f6e4e17bd54fdaf1819a8a2@[729d64003e7740cabb650f8f6aea4af6(127.1.76.194:60468),7a2e5f9b2be9497fadc30b81a6a50b24(127.1.76.19
20:05:05.409 [DEBUG - New I/O worker #17] (AsyncKuduScanner.java:934) AsyncKuduScanner$Response(scannerId = "", data = RowResultIterator for 7314 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.ITClient-1522206255318, tablet=d78c
20:05:05.409 [INFO - Thread-4] (ITClient.java:397) New row count 90114
20:05:05.414 [DEBUG - New I/O worker #17] (AsyncKuduScanner.java:934) AsyncKuduScanner$Response(scannerId = "c230614ad13e40478254b785995d1d7c", data = RowResultIterator for 27600 rows, more = true, responseScanTimestamp = 6234957022413987840) for scanner
20:05:05.414 [DEBUG - New I/O worker #17] (AsyncKuduScanner.java:447) Scanner "c230614ad13e40478254b785995d1d7c" opened on d78cb5506f6e4e17bd54fdaf1819a8a2@[729d64003e7740cabb650f8f6aea4af6(127.1.76.194:60468),7a2e5f9b2be9497fadc30b81a6a50b24(127.1.76.19
20:05:05.419 [DEBUG - New I/O worker #17] (AsyncKuduScanner.java:934) AsyncKuduScanner$Response(scannerId = "", data = RowResultIterator for 27600 rows, more = true) for scanner KuduScanner(table=org.apache.kudu.client.ITClient-1522206255318, tablet=d78c
20:05:05.420 [DEBUG - New I/O worker #17] (AsyncKuduScanner.java:934) AsyncKuduScanner$Response(scannerId = "", data = RowResultIterator for 7342 rows, more = false) for scanner KuduScanner(table=org.apache.kudu.client.ITClient-1522206255318, tablet=d78c
20:05:05.421 [ERROR - Thread-4] (ITClient.java:134) Row count unexpectedly decreased from 90114to 62542