[ARROW-9660] [C++] IPC - dictionaries in maps

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.0.0
• Fix Version/s: 2.0.0
• Component/s: C++
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/25719
• Assignee: Antoine Pitrou
• Reporter: Pierre Belzile
• Created: 05/Aug/20 23:53
• Updated: 11/Jan/23 08:08
• Resolved: 27/Aug/20 16:10

Description

I created the following record batch which has a single column with a type of map<dict, string> where dict is defined as: dict<int8,string>:

 

arrow::MapBuilder map_builder(arrow::default_memory_pool(),
    std::make_shared<arrow::StringDictionaryBuilder>(),
    std::make_shared<arrow::StringBuilder>());
auto key_builder = 
    dynamic_cast<arrow::StringDictionaryBuilder *>(map_builder.key_builder());
auto item_builder = 
    dynamic_cast<arrow::StringBuilder *>(map_builder.item_builder());

// Add a first row with k<i>=v<i> for i 0..14;
ASSERT_OK(map_builder.Append());
for (int i = 0; i < 15; ++i) {
  ASSERT_OK(key_builder->Append("k" + std::to_string(i)));
  ASSERT_OK(item_builder->Append("v" + std::to_string(i)));
}
// Add a second row with k<i>=w<i> for i 0..14;
ASSERT_OK(map_builder.Append());
for (int i = 0; i < 15; ++i) {
  ASSERT_OK(key_builder->Append("k" + std::to_string(i)));
  ASSERT_OK(item_builder->Append("w" + std::to_string(i)));
}
std::shared_ptr<arrow::Array> array;
ASSERT_OK(map_builder.Finish(&array));
std::shared_ptr<arrow::Schema> schema = 
    arrow::schema({arrow::field("s", array->type())});
std::shared_ptr<arrow::RecordBatch> batch = 
    arrow::RecordBatch::Make(schema, array->length(), {array});

When one attempts to send this in a round trip IPC:

1. On IpcFormatWriter::Start(): The memo records one entry for field_to_id and id_to_type_ where the dict id = 0.
2. On IpcFormatWriter::CollectDictionaries: The memo records a new entry for field_to_id and id_to_type with id=1 and also records in id_to_dictionary_. At this point we have 2 entries with the entry id=0 having no associated dict.
3. On IpcFormatWriter;:WriteDictionaries: It writes the dict with entry = 1

When reading:

1. GetSchema eventually gets to the nested dictionary in FieldFromFlatBuffer
2. The recovered dict id is 0.
3. This adds to the memo the field_to_id and id_to_type with id = 0
4. My round trip code calls "ReadAll".
5. RecordBatchStreamReaderImpl::ReadNext attempts to load the initial dicts
6. It recovers id = 1
7. The process aborts because id = 1 is not in the memo: dictionary_memo->GetDictionaryType(id, &value_type)

A similar example with a dict inside a "struct" worked fine and only used dict id = 0. So it looks like something wrong when gathering the schema for the map. Unless I did not construct the map correctly?

 

Issue Links

• is duplicated by: ARROW-8749 [C++] IpcFormatWriter writes dictionary batches with wrong ID
• links to: GitHub Pull Request #7992