Critical Description
We observed the following incorrect behavior of reporting errors in a KuduSession when using AUTO_FLUSH_BACKGROUND:
1. A client writes 2363 rows to a table, and 39 of those rows have PKs that already exist so should return duplicate key error. All rows are applied to the session.
2. The client calls Flush().
expectation: All outstanding operations should wait until they complete successfully or an error occurs, in which case the error is reported in GetPendingErrors()
3. The client calls KuduSession::GetPendingErrors() and observes 38 errors.
I identified this using logging from an Impala daemon:
Impala logs some information, as well as output from the Kudu client. In it, I can see that batcher.cc reports 38 errors, then Impala calls GetPendingErrors() and indicates there were 38 errors, and this Impala logging only occurs after the last Flush() call. However, during the course of Impala handling those 38 errors, batcher.cc reports one more error but this was already after the last Flush() call so Impala wouldn't expect to have to call GetPendingErrors() again.
Here's the relevant log snippet:
... #### total of 38 of these batcher.cc errors, only 1 shown I1109 01:04:13.847666 23365 batcher.cc:757] Error on op INSERT int32 id=94, bool bool_col=true, int8 tinyint_col=4, int16 smallint_col=4, int32 int_col=4, int64 bigint_col=40, float float_col=4.4, double double_col=40.4, string date_string_col=01/10/09, string string_col=4, string timestamp_col=2009-01-10 01:34:04.110000000, int32 year=2009, int32 month=1: code: ALREADY_PRESENT message: "key already present" #### Impala logging occurs after the Flush() call returns, at which point it fetches the pending errors, logs them and shuts down the session. There are a total of 38 of these errors: I1109 01:04:13.848956 23325 runtime-state.cc:208] Error from query 3840a50eed6a1429:88d4a4e700000000: Key already present in Kudu table 'impala::test_kudu_crud_fe258600.kudu_test_tbl'. I1109 01:04:13.849195 23325 runtime-state.cc:208] Error from query 3840a50eed6a1429:88d4a4e700000000: Key already present in Kudu table 'impala::test_kudu_crud_fe258600.kudu_test_tbl'. I1109 01:04:13.849400 23325 runtime-state.cc:208] Error from query 3840a50eed6a1429:88d4a4e700000000: Key already present in Kudu table 'impala::test_kudu_crud_fe258600.kudu_test_tbl'. I1109 01:04:13.849617 23325 runtime-state.cc:208] Error from query 3840a50eed6a1429:88d4a4e700000000: Key already present in Kudu table 'impala::test_kudu_crud_fe258600.kudu_test_tbl'. I1109 01:04:13.849817 23325 runtime-state.cc:208] Error from query 3840a50eed6a1429:88d4a4e700000000: Key already present in Kudu table 'impala::test_kudu_crud_fe258600.kudu_test_tbl'. I1109 01:04:13.850013 23325 runtime-state.cc:208] Error from query 3840a50eed6a1429:88d4a4e700000000: Key already present in Kudu table 'impala::test_kudu_crud_fe258600.kudu_test_tbl'. I1109 01:04:13.850211 23325 runtime-state.cc:208] Error from query 3840a50eed6a1429:88d4a4e700000000: Key already present in Kudu table 'impala::test_kudu_crud_fe258600.kudu_test_tbl'. I1109 01:04:13.850436 23325 runtime-state.cc:208] Error from query 3840a50eed6a1429:88d4a4e700000000: Key already present in Kudu table 'impala::test_kudu_crud_fe258600.kudu_test_tbl'. I1109 01:04:13.850641 23325 runtime-state.cc:208] Error from query 3840a50eed6a1429:88d4a4e700000000: Key already present in Kudu table 'impala::test_kudu_crud_fe258600.kudu_test_tbl'. I1109 01:04:13.850831 23325 runtime-state.cc:208] Error from query 3840a50eed6a1429:88d4a4e700000000: Key already present in Kudu table 'impala::test_kudu_crud_fe258600.kudu_test_tbl'. I1109 01:04:13.851024 23325 runtime-state.cc:208] Error from query 3840a50eed6a1429:88d4a4e700000000: Key already present in Kudu table 'impala::test_kudu_crud_fe258600.kudu_test_tbl'. I1109 01:04:13.851232 23325 runtime-state.cc:208] Error from query 3840a50eed6a1429:88d4a4e700000000: Key already present in Kudu table 'impala::test_kudu_crud_fe258600.kudu_test_tbl'. I1109 01:04:13.851449 23325 runtime-state.cc:208] Error from query 3840a50eed6a1429:88d4a4e700000000: Key already present in Kudu table 'impala::test_kudu_crud_fe258600.kudu_test_tbl'. I1109 01:04:13.851647 23325 runtime-state.cc:208] Error from query 3840a50eed6a1429:88d4a4e700000000: Key already present in Kudu table 'impala::test_kudu_crud_fe258600.kudu_test_tbl'. I1109 01:04:13.851841 23325 runtime-state.cc:208] Error from query 3840a50eed6a1429:88d4a4e700000000: Key already present in Kudu table 'impala::test_kudu_crud_fe258600.kudu_test_tbl'. I1109 01:04:13.852088 23325 runtime-state.cc:208] Error from query 3840a50eed6a1429:88d4a4e700000000: Key already present in Kudu table 'impala::test_kudu_crud_fe258600.kudu_test_tbl'. #### the batcher.cc reports one more error mid way through that will never be seen by Impala: I1109 01:04:13.852166 23365 batcher.cc:757] Error on op INSERT int32 id=97, bool bool_col=false, int8 tinyint_col=7, int16 smallint_col=7, int32 int_col=7, int64 bigint_col=70, float float_col=7.7, double double_col=70.7, string date_string_col=01/10/09, string string_col=7, string timestamp_col=2009-01-10 01:37:04.260000000, int32 year=2009, int32 month=1: code: ALREADY_PRESENT message: "key already present" I1109 01:04:13.852283 23325 runtime-state.cc:208] Error from query 3840a50eed6a1429:88d4a4e700000000: Key already present in Kudu table 'impala::test_kudu_crud_fe258600.kudu_test_tbl'. ...