Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-17394

AvroSerde is regenerating TypeInfo objects for each nullable Avro field for every row

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.0, 3.0.0
    • Fix Version/s: 3.0.0
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The following methods in AvroDeserializer keeps regenerating TypeInfo objects for every nullable field in a row.

      This is happening in the following methods.

      private Object deserializeNullableUnion(Object datum, Schema fileSchema, Schema recordSchema) throws AvroSerdeException {
      // elided
      line 312:  return worker(datum, fileSchema, newRecordSchema,
                  SchemaToTypeInfo.generateTypeInfo(newRecordSchema, null));
      }
      ..
      private Object deserializeSingleItemNullableUnion(Object datum, Schema Schema recordSchema)
      // elided
      line 357: return worker(datum, currentFileSchema, schema,
            SchemaToTypeInfo.generateTypeInfo(schema, null));
      

      This is really bad in terms of performance. I'm not sure why didn't we use the TypeInfo we already have instead of generating again for each nullable field. If you look at the worker method which calls the method deserializeNullableUnion the typeInfo corresponding to the nullable field column is already determined.
      Moreover the cache in SchemaToTypeInfo class does not help in nullable Avro records case as checking if an Avro record schema object already exists in the cache requires traversing all the fields in the record schema.

      I've attached profiling snapshot which shows maximum time is being spent in the cache.

      One way of fixing this IMO might be to make use of the column TypeInfo which is already passed in the worker method.

        Attachments

        1. AvroSerDe.nps
          15 kB
          Ratandeep Ratti
        2. AvroSerDeUnionTypeInfo.png
          88 kB
          Ratandeep Ratti
        3. HIVE-17394.1.patch
          4 kB
          Anthony Hsu

          Issue Links

            Activity

              People

              • Assignee:
                erwaman Anthony Hsu
                Reporter:
                rdsr Ratandeep Ratti
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: