[IMPALA-2729] Support default values for DECIMAL in Avro.

• Type: New Feature
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: Impala 2.0, Impala 2.1, Impala 2.2, Impala 2.3.0
• Component/s: Backend
• Labels: ramp-up usability
• Target Version: Product Backlog
• Assignee: Unassigned
• Reporter: Alexander Behm
• Created: 01/Dec/15 08:37
• Updated: 30/Oct/18 17:41

Description

Impala may fail to scan Avro files of a table that has default values specified on DECIMAL columns.

The following criteria must be met for hitting this bug:
1. The Impala table was created from an Avro schema that has a default value on a DECIMAL field
2. There is an Avro file storing data of that table that does not have the DECIMAL field declared as part of the table schema from 1

When querying such a table Impala will return the following error

Field 'decimalFieldName' is missing from file and default values of type DECIMAL are not yet supported.

The relevant code can be found in hdfs-avro-scanner.cc:

Status HdfsAvroScanner::WriteDefaultValue(
    SlotDescriptor* slot_desc, avro_datum_t default_value, const char* field_name) {
  if (avro_header_->template_tuple == NULL) {
    avro_header_->template_tuple = template_tuple_ != NULL ?
        template_tuple_ : scan_node_->InitEmptyTemplateTuple(*scan_node_->tuple_desc());
  }
  switch (default_value->type) {
    case AVRO_BOOLEAN: {
      // We don't call VerifyTypesMatch() above the switch statement so we don't want to
      // call it in the default case (since we VerifyTypesMatch() can't handle every type
      // either, and we want to return the correct error message).
      RETURN_IF_ERROR(VerifyTypesMatch(slot_desc, default_value));
      int8_t v;
      if (avro_boolean_get(default_value, &v)) DCHECK(false);
      RawValue::Write(&v, avro_header_->template_tuple, slot_desc, NULL);
      break;
    }
    case AVRO_INT32: {
      RETURN_IF_ERROR(VerifyTypesMatch(slot_desc, default_value));
      int32_t v;
      if (avro_int32_get(default_value, &v)) DCHECK(false);
      RawValue::Write(&v, avro_header_->template_tuple, slot_desc, NULL);
      break;
    }
...
    <--- case for AVRO_DECIMAL not handled
    default:
      return Status(TErrorCode::AVRO_UNSUPPORTED_DEFAULT_VALUE, field_name,
          avro_type_name(default_value->type));