[ARROW-8749] [C++] IpcFormatWriter writes dictionary batches with wrong ID

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.16.0, 0.17.0
• Fix Version/s: 2.0.0
• Component/s: C++
• External issue URL: https://github.com/apache/arrow/issues/24899
• Assignee: Antoine Pitrou
• Reporter: David Li
• Created: 09/May/20 20:46
• Updated: 11/Jan/23 08:02
• Resolved: 29/Aug/20 19:37

Description

IpcFormatWriter assigns dictionary IDs once when it writes the schema message. Then, when it writes dictionary batches, it assigns dictionary IDs again because it re-collects dictionaries from the given batch. So for example, if you have 5 dictionaries, the first dictionary will end up with ID 0 but be written with ID 5.

For example, this will fail with "'_error_or_value11.status()' failed with Key error: No record of dictionary type with id 9"

TEST_F(TestMetadata, DoPutDictionaries) {
  ASSERT_OK_AND_ASSIGN(auto sink, arrow::io::BufferOutputStream::Create());
  std::shared_ptr<Schema> schema = ExampleDictSchema();
  BatchVector expected_batches;
  ASSERT_OK(ExampleDictBatches(&expected_batches));
  ASSERT_OK_AND_ASSIGN(auto writer, arrow::ipc::NewStreamWriter(sink.get(), schema));
  for (auto& batch : expected_batches) {
    ASSERT_OK(writer->WriteRecordBatch(*batch));
  }
  ASSERT_OK_AND_ASSIGN(auto buf, sink->Finish());
  arrow::io::BufferReader source(buf);
  ASSERT_OK_AND_ASSIGN(auto reader, arrow::ipc::RecordBatchStreamReader::Open(&source));
  AssertSchemaEqual(schema, reader->schema());
  for (auto& batch : expected_batches) {
    ASSERT_OK_AND_ASSIGN(auto actual, reader->Next());
    AssertBatchesEqual(*actual, *batch);
  }
}

Issue Links

• duplicates: ARROW-9660 [C++] IPC - dictionaries in maps