[IMPALA-1254] parquet writer will incorrectly add a value twice when max dictionary size is reached

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 1.4
• Fix Version/s: Impala 2.1
• Assignee: Daniel Hecht
• Reporter: Daniel Hecht
• Created: 15/Sep/14 23:17
• Updated: 11/Nov/14 17:23
• Resolved: 11/Nov/14 17:23

Description

EncodeValue returns false when the dictionary num entries limit has been reached, even though the value was added. Then, AppendRow() will loop back around and re-add that value.

   if (current_encoding_ == Encoding::PLAIN_DICTIONARY) {
      *bytes_needed = dict_encoder_->Put(*reinterpret_cast<T*>(value));
      *bytes_added += *bytes_needed;

      // If the dictionary contains the maximum number of values, switch to plain
      // encoding.  The current dictionary encoded page is written out.
      if (dict_encoder_->num_entries() == MAX_DICTIONARY_ENTRIES) {
        *bytes_added += FinalizeCurrentPage();
        current_encoding_ = Encoding::PLAIN;
        return false;                           <==== HERE
      } else {