[HIVE-14792] AvroSerde reads the remote schema-file at least once per mapper, per table reference.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.1, 2.1.0
• Fix Version/s: 2.2.1, 2.4.0, 3.0.0
• Labels: TODOC2.2 TODOC2.4
• Assignee: Aihua Xu
• Reporter: Mithun Radhakrishnan
• Created: 19/Sep/16 22:51
• Updated: 23/May/18 00:00
• Resolved: 12/Mar/18 22:01

Description

Avro tables that use "external" schema files stored on HDFS can cause excessive calls to FileSystem::open(), especially for queries that spawn large numbers of mappers.

This is because of the following code in AvroSerDe::initialize():

AvroSerDe.java

public void initialize(Configuration configuration, Properties properties) throws SerDeException {
// ...
    if (hasExternalSchema(properties)
        || columnNameProperty == null || columnNameProperty.isEmpty()
        || columnTypeProperty == null || columnTypeProperty.isEmpty()) {
      schema = determineSchemaOrReturnErrorSchema(configuration, properties);
    } else {
      // Get column names and sort order
      columnNames = Arrays.asList(columnNameProperty.split(","));
      columnTypes = TypeInfoUtils.getTypeInfosFromTypeString(columnTypeProperty);

      schema = getSchemaFromCols(properties, columnNames, columnTypes, columnCommentProperty);
         properties.setProperty(AvroSerdeUtils.AvroTableProperties.SCHEMA_LITERAL.getPropName(), schema.toString());
    }
// ...
}

For tables using avro.schema.url, every time the SerDe is initialized (i.e. at least once per mapper), the schema file is read remotely. For queries with thousands of mappers, this leads to a stampede to the handful (3?) datanodes that host the schema-file. In the best case, this causes slowdowns.

It would be preferable to distribute the Avro-schema to all mappers as part of the job-conf. The alternatives aren't exactly appealing:

1. One can't rely solely on the column.list.types stored in the Hive metastore. (HIVE-14789).
2. avro.schema.literal might not always be usable, because of the size-limit on table-parameters. The typical size of the Avro-schema file is between 0.5-3MB, in my limited experience. Bumping the max table-parameter size isn't a great solution.

If the avro.schema.file were read during query-planning, and made available as part of table-properties (but not serialized into the metastore), the downstream logic will remain largely intact. I have a patch that does this.

Issue Links

• is depended upon by: HIVE-14794 HCatalog support to pre-fetch schema for Avro tables that use avro.schema.url.